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1  Oth  Annual  Systems  Engineering  Conference 
“Focusing  on  Improving  Performance  of  Defense  Systems  Programs ” 

San  Diego,  CA 
22-25  October  2007 
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Agenda 


Tuesday.  23  October  2007 

Keynote  Addresses: 

•  Hon  James  Finley.  Deputy  Under  Secretary  of  Defense,  Acquisition  and  Technology 

•  Hon  Charles  McOueary.  Director,  Operational  Test  and  Evaluation 

Plenary  Session:  Executive  Panel: 

•  Mr.  Mark  Schaeffer.  Director,  Office  of  Under  Secretary  of  Defense  of  Acquisition,  Technology  and  Logistics;  Director,  Systems  and  Software  Engineering 

Panelists: 

•  Mr.  Terry  Jaggers .  SAF/AQR  -  Science,  Technology,  and  Engineering 

•  Mr.  Carl  Siel.  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy  Research,  Development  and  Acquisition 

•  Mr.  Doug  Wiltsie.  HODA.  OASA  (ALT) 


Track  1 

•  “Update:  OSD  Systems  Engineering  Revitalization  Efforts,”  Col  Richard  Hoeferkamp,  USAF 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  (DoD)  System  Development  Programs”,  Mr.  A1  Mink,  SRA  International 

•  “Tools  and  Resources  to  Enable  Systems  Engineering  Improvement,”  Mr.  Michael  Kutch,  SPAWAR  Systems  Center  Charleston 

•  “Sound  Systems  Engineering  Assures  Proper/Early  Producibility”,  Dr.  Thomas  Christian,  Aeronautical  Systems  Center 

•  “Realization  of  Systems  Engineering  For  the  Future”,  Ms.  Karen  Bausman,  AF  Center  for  Systems  Engineering 

Track  2 

•  “Developmental  Test  &  Evaluation  Policy  Vectors”,  Ms.  Darlene  Mosser-Kemer  OUSD  (AT&L) 

•  “Test  Strategy  Done  Early  Drives  Test  Planning  and  Successful  Testing”,  Mr.  William  Lyders,  AS  SETT,  Inc. 

•  “Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate  Evaluation”,  Mr.  Joseph  Tribble, _AVW_“Implementing  a  Systems  Engineering 
Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

•  “Joint  Safety  Review  Process  Study”,  Ms.  Paige  Ripini,  Booz  Allen  Hamilton 

Track  3 

•  “DoD  Systemic  Root  Cause  Analysis”,  Mr.  Dave  Castellano,  OUSD  (AT&L) 

•  “Applying  Systems  Engineering  During  Pre-Acquisition  Activities”,  Lt  Col  Mark  Wilson,  USAF 

•  “Reforming  the  DoD  Acquisition  Process — A  Systems  Engineering  Approach”,  Mr.  Stephan  Ward,  U.S.  Air  Force 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  System  Development  Programs”,  Mr.  Alan  Mink,  SRA  International 

Track  4 

•  “Environment,  Safety,  and  Occupational  Health  (ESOH) — Design  Considerations  to  Support  Sustainability  and  Readiness”,  Ms.  Patricia  Huheey,  ODUSD 
(I&E) 

•  “Real-Time  Diagnostics  for  High  Availability  Systems”,  Mr.  Edward  Beck,  Computer  Sciences  Corporation 

•  “Sparing  Satellites  Comparative  Strategies  of  On-orbit  &  In-factory  Storage”,  Mr.  James  Mazzei,  The  Aerospace  Corporation 

Track  5 

•  “Acquisition  M&S  Master  Plan  Implementation  Status”,  Mr.  Michael  Truelove,  SAIC 

•  “Establish  M&S-Related  Guidelines  for  Solicitations,  Source  Selections,  and  Contracting”,  Mr.  Michael  Truelove,  SAIC 

•  “Modeling  and  Simulation  Resource  Reuse  Business  Model”,  Mr.  Dennis  Shea,  Center  for  Naval  Analyses 
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•  “Modeling  and  Simulation  Support  Plan”,  Mr.  David  Henry,  Lockheed  Martin 

•  “Modeling  and  Simulation  Education  for  the  Acquisition/T&E  Workforce:  Requirements  Analysis”,  Mr.  David  Olwell,  NPS 
Track  6 

•  “The  Use  of  Enterprise  Architecture  to  Support  the  Development  of  the  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  David  Synder,  The 
MITRE  Corp. 

•  “Complex  Systems  of  Systems:  The  Dual  Challenge”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 

•  “Systems  Engineering  in  the  Cognitive  and  Social  Domains  of  Net  Centric  Operations”,  Dr.  Abe  Meilich,  Lockheed  Martin 


Track  7 

•  “Simplifying  &  Scaling  Engineering  Processes:  Unifying  Business  Units  and  Engineering  Disciplines”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “NAVAIR  Systems  Engineering  Revitalization”,  Mr.  Michael  Gaydar,  Department  of  Navy,  NAVAIR 

•  “Integrating  Engineering  Project  Management  and  Product  Development  Processes”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “Engineering  for  System  Assurance — Legacy,  Life  Cycle,  Leadership”,  Mr.  Paul  Croll,  Computer  Sciences  Corporation 

Track  8 

o  “DoD  Software  Engineering  and  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software  Engineering 
o  “The  Integrated  Software  and  Systems  Engineering  Curriculum  Project:  Creating  a  Reference  Curriculum  for  Graduate  Software  Engineering 
Education”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software 
o  “Requirements  for  a  Chief  Software  Engineer  in  a  DoD  Acquisition  Agency”,  Mr.  A1  Florence,  The  MITRE  Corporation 
o  “Developing  an  Integrated  Process  Methodology  for  Interim  Software  Releases”,  Mr.  Tim  Woods,  Southern  Methodist  University 

Wednesday.  24  October  2007 
Track  1 

o  “Change  Management  of  UML-Based  Systems  Engineering  Artefacts”,  Mr.  David  Price,  Eurostep 
o  “A  Day  in  the  Life  of  a  Verification  Requirement”,  Mr.  Stephen  Scukanec,  Northrop  Grumman  Corporation 
o  “How  to  Measurably  Improve  Your  Requirements”,  Mr.  Timothy  Olson,  Lean,  Solutions  Institute,  Inc. 
o  “Case  Studies:  A  Common  Language  Between  Engineers  and  Managers”,  Capt  DeWitt  Latimer,  USAF 
o  “A  Strategy  for  Improved  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

°  “Discussion  of  the  U.S.  Army  RDECOM  APS  Objective  Trade  Study”,  Mr.  Frank  Salvatore,  High  Performance  Technologies,  Inc. 
o  “Program  Support  Review  Deep  Dive”,  Mr.  Peter  Nolte,  OUSD/SSE 

Track  2 

o  “An  Update  on  the  DT&E  Committee’s  Recommended  Policy  Changes  to  DoD  5000”,  Col  Richard  Stuckey,USAF,OUSD  (AT&L)/SSE/  DT&E 
o  “System  Test  and  Evaluation  in  the  DARPA  Immune  Building  Demonstration  Program”,  Mr.  Mark  Saxon,Battelle 

o  “  Modeling  and  Simulation  in  the  Navy  Warfare  Systems  Test  &  Evaluation  Enterprise”,  Ms.  Shala  Malone, Navy  Program  Executive  Office  Integrated 
Warfare 

o  “Joint  Mission  Environment  Test  Capability  (JMETC)”,  Mr.  Richard  Lockhart,  Test  Resource  Management  Center 
o  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina 
o  Research  Authority  (SCRA 

o  “Do  it  right,  do  it  early;  Do  it  early,  do  it  right” — Considerations  for  the  Early  Stages  of  Concept,  System,  and  Systems  of-Systems  Definition”,  Mr.  Jeff 
Loren,  MTC  Technologies,  Inc.  (SAF/AQRE) 

o  “Applications  of  Systems  Engineering  to  Pre-Milestone  A  Projects”,  Ms.  Lori  Zipes, Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  in  a  Systems  of  Systems  Environment  -  Defense  Update”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 
o  “ASW  System-of- Systems  Engineering  Analysis  Applied  in  an  LCS  ASW  Integration  Pilot  Project”,  Mr.  G.  Richard  Thompson,  JHU/APL 


Track  3 

o  “Systems  Engineering  Plan  Preparation  Guide  Update”,  Mr.  Chester  Bracuto,  OSD/AT&L/A&T/SSE/ED 
o  “  Toward  a  Unified  Systems  Engineering  Plan”,  Mr.  Robert  Scheurer,  Boeing  Integrated  Defense  Systems 
o  “Integrating  Risk  &  Knowledge  Management”,  Mr.  David  Lengyel,  NASA 

o  “Systems  Engineering  and  Program  Management — How  Different  are  They?”,  Ms.  Lori  Zipes,  Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  Analysis  to  Improve  Concept  Development  of  Complex  Defense  Systems”,  Mr.  Michael  Harper,  SPAWAR  Systems  Center 
Charleston 

o  “  The  Joint  Partnership  Between  Program  Management  and  Systems  Engineering”,  Mr.  Samuel  Son,  The  Boeing  Company 
o  “Lockheed  Martin  Aeronautics  Company  Approach  to  Solving  Systemic  Development  Program  Issues”,  Mr.  John  Weaver,  Lockheed  Martin 
Aeronautics  Company 

o  “Airborne  Signals  Intelligence  Payload  (ASIP)  Program  Integrated  Risk  Management”,  Mr.  Danial  Bolek,  USAF 
o  “Improvements  to  the  Risk  Management  Process”,  Mr.  Doug  Atkinson,  USAF 
o  “Integrated  Risk  and  Earned  Value  Management”,  Mr.  Paul  Davis,  Northrop  Grumman 

o  “Application  of  Risk  Management  Practices  to  NNSA  Tritium  Readiness  Subprogram”, Mr.  Sham  Shete,  Washington  Savannah  River  Co. 

Track  4 

o  “Defining  Lean  Service  and  Maintenance  Processes”,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc. 

o  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”,  Mr.  Patrick  Cumby,  VectorCSP,  L 
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“Asset-Based  PBL  for  Navy  Warships  -  A  Case  Study  for  LCS  Class  Ships”,  Mr.  Michael  Mahon,  Lockheed  Martin 
o  “Integrated  Diagnostics  Closed  Loop  Data  System  (At  the  Point  of  Use)  (Support  Systems  Knowledge  Engineering  Enhances  Traditional  Support 
Equipment  Systems  Engineering),”  Mr.  Steven  Head,  Boeing 
o  “Aging  Aircraft  Sustainment  with  Non-Standard  Engineering”,  Ms.  Kendal  Hinton,  Georgia  Tech  Research  Institute 
o  “Maintaining  System  Viability  for  the  Long  Term”,  Mr.  Peter  Henry,  BAE  Systems  Land  and  Armament 
o  “C-17  Program  Applies  Systems  Engineering  to  a  Large  Improvement  Project”,  Mr.  Brent  Theodore, The  Boeing  Company 

Track  5 

o  “ Advancing  the  FEDEP  for  Simulation  Based  Acquisition ”,  Dr.  Katherine  Morse,  SAIC 

o  “Acquisition  M&S  Community  Sponsored  M&S  Project:  Standardized  Documentation  for  Verification,  Validation,  and  Accreditation”,  Mr.  Kevin 
Charlow,  (paper)  (slides)  Space  and  Naval  Warfare  Systems  Center  Charleston 
o  “A  Methodology  for  Assessing  &  Prioritizing  the  Risks  Associated  with  the  Level  of  Verification,  Validation  and  Accreditation  (VV&A)  of  Models 
and  Simulations”,  Dr.  James  Elele,  U.S.  Navy 

o  “Experiences  in  Applying  SysML  to  Develop  Interoperable  Torpedo  Modeling  and  Simulation  Components”,  Mr.  Thomas  Haley,  Naval  Undersea 
Warfare  Center 

o  “Unifying  Systems  Engineering  Simulations”,  Mr.  Ryan  O’ Grady,  Cybernet  Systems  Corporation 
o  “Information  Modeling  for  Systems  Integration”,  Ms.  Claudia  Rose,BBII 

o  “Simulation  Supported  Decision  Making”,  (slidesl)  (slides  2)  Mr.  Gene  Allen,  MSC  Software  Corporation 
Track  6 

o  “Achieving  Agility  in  Cyberspace”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 
o  “Application  of  Autonomic  Agents  for  Global  Information  Grid”,  Mr.  David  Cox,  University  of  Arizona 
o  “Architecture-Based  Concept  Evaluation  in  Support  of  JCIDS”,  Dr.  David  Jacques,  Air  Force  Institute  of  Technology 
o  “System  of  Systems  Implications  for  Operational  Test”,  Dr.  John  Colombi,  Air  Force  Institute  of  Technology 
°  “Case  Study:  Net  Centric  Mission  Thread  Modeling  and  Analysis”,  Dr.  Prem  Jain,  MITRE 

o  “Quantitative  Comparison  of  Alternative  Designs  for  a  JC3M  System”,  Mr.  Gregory  Miller,  Naval  Postgraduate  School 
o  “Advanced  Net  Centric  Simulation  for  Aerospace  Command  and  Control”,  Ms.  Kimberly  Kendall,  753d  ELSG/NEM,  ESC,  USAF 


Track  7 

o  “CMMI— Next  Steps”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

o  “CMMI  Instructional  Challenges  to  Systems  Engineers  in  Small  Settings”,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 
“FISMA  Operational  Controls  and  Their  Relationship  to  Process  Maturity”,  Ms.  Rhonda  Henning,  Harris  Corporation 

o  “Executing  a  Successful  CMMI  Maturity  Level  3  SCAMPI  For  SPAWAR  Systems  Center  Charleston”,  Mr.  Michael  Kutch,  SPAWAR  Systems  Center 
Charleston 

o  “CMMI  for  Services:  Re-Introducing  the  CMMI  for  Services  Constellation”,  Mr.  Craig  Hollenbach,  Northrop  Grumman  Corporation 

o  “How  to  Paint  a  Room:  The  Role  of  Specs  &  Standards  in  SE”,  Mr.  Robert  Kuhnen,  USAF 

o  “Continuous  Improvement  at  the  Organization,  Team,  and  Individual  Levels  — Lessons  Learned  Integrating  CMM,TSP,  and  PSP  and  Why  All  Three 
are  Needed”,  Mr.  Girish  Seshagiri, Advanced  Information  Services,  Inc 

o  “Addressing  Environment,  Safety  and  Occupational  Health  Issues  for  the  Mine  Resistant  Ambush  Protected  (MRAP)  Vehicle  Program”,  Ms.  Jennifer 
Malone,  EG&G 

o  “Anatomy  of  an  Award  Winning  Safety  Program:  A  Case  Study  of  the  SSGN  OHIO  Class  Conversion  Safety  Program”,  Mr.  Ricky  Milnarik,  Electric 
Boat  Corporation 

o  “The  Safety  of  Unmanned  Systems:  The  Process  Used  to  Develop  Safety  Precepts  for  Unmanned  Systems”,  Mr.  Mike  Demmick,  NOSSA 
Track  8 


■  “Defining  Software  Component  Specifications:  An  Open  Approach”,  Mr.  Kenneth  Klein, Computer  Sciences  Corporation 

■  “System  Engineering  and  Software  Exception  Handling  (SEH)”,  Mr.  Herbert  Hecht,  SoHaR  Incorporated 

■  “A  Convergence  of  Technologies  for  Better  Software  NOW!”,  Ms.  Dorothy  Acton,  Lockheed  Martin  IS&GS 

■  “Identifying  Acquisition  Patterns  of  Failure  Using  System  Archetypes”,  Mr.  William  Novak,  Software  Engineering  Institute 

■  “Revitalizing  Education  and  Training  in  Systems  Engineering” ,  Dr.  Don  Gelosh,  Department  of  Defense,  OSD(AT&L)/SSE/ED 

■  “Customer-Driven,  Partnership-Based  Systems  Engineering  Education  and  Training”,  Mr.  Jerrell  Stracener,  Southern  Methodist  University 

■  “Application  of  Systems  Engineering  Principles  in  the  Design  of  Acquisition  Workforce  Curricula”,  (paper)  (slidel  Dr.  David  Olwell.  Naval 
Postgraduate  School 


Thursday.  25  October  2007 
Track  1 


■  “USAF  Type  Certification  of  Commercial  Derivative  Aircraft”,  Mr.  Thomas  Morgan,  USAF 

■  “Global  Positioning  System  Case  Study”,  Mr.  Randall  Bullard,  Air  Force  Center  for  Systems  Engineering 

Track  2 

■  “Implementing  the  Technology  Maturity  Vector”,  Mr.  Joseph  Terlizzese,  Systems  Engineering  Support  Office 

■  “Technology  Readiness  Assessments;  Milestone  B  Certification  Requirement  for  Technologies  to  be  Demonstrated  in  a  Relevant  Environment”, 
Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analyses 

■  “Meeting  Enterprise  System  Engineering  Challenges  for  the  U.S.  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  Jerry  Friedman, 
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The  MITRE  Corporation 

■  “Sensor  Resource  Allocation  as  a  Driver  in  System  Concept  Development”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  3 

■  “Managing  Requirements  to  Manage  Scope  in  the  Case  of  MUOS”.  Ms.  Christy  Howard.  Maxim  Systems,  Inc. 

■  “Organizational  Leadership  and  Management  Dynamics  for  Technical  Execution  in  Acquisition  Programs”.  Mr.  Francis  Sisti.  Aerospace 
Corporation 

■  “C-17  Systems  Engineering  Process  to  Prioritize  Material  Improvement  Program  (MIP)  Projects”,  Mr.  Thomas  Condron,  USAF  (516 
AESG/ASC) 

■  “How  to  Talk  to  a  Program  Manager”.  Dr.  John  Mishler.  Software  Engineering  Institute 

■  “U.S.  Department  of  Defense  (DoD)  Approach  to  Best  Practices:  Building  Evidence  for  Practice  Selection  Based  on  Real  Experiences”,  Dr. 
Forrest  Shull,  Fraunhofer  Center  Maryland 

Track  4 

■  “Strategic  Focus:  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  (VE)”,  Dr.  Danny  Reed,  Institute  for  Defense  Analyses 

■  “Progress  Toward  an  Empirical  Relationships  Between  Reliability  Investments  and  Life-Cycle  Support  Cost”,  Dr.  James  Forbes,  LMI 

■  “Innovation  Strategies  for  Affordable  Readiness”.  Mr.  Tom  Choinski.  Naval  Undersea  Warfare  Center 

■  “Implementing  a  Systems  Engineering  Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

■  “Asset-Based  PBL  for  Navy  Warships  -  A  case  study  for  LCA  Class  Ships”,  Mike  Mahon 

■  “The  Deployment  Readiness  Service:  The  Challenges  of  Implementing  a  Service  Oriented  Architecture  in  a  Legacy  System  Environment”,  Mr. 
George  Dalton,  USAF 

Track  5 

■  “Aircraft  Flight  Simulator  Acceptance  Criteria”,  Mr.  Dean  Carico,  NAWCAD  PAX 

■  “Computer  Modeling  to  Solve  Problems  with  the  T-38  Propulsion  Modernization  Program”,  Mr.  Randall  Wimer,  USAF  -  FVB 

■  “Efficacy  of  Modeling  &  Simulation  in  Defense  Life  Cycle  Engineering”,  Mr.  Donald  Cox,  Raytheon  Missile  Systems 

■  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”.  Mr.  Patrick  Cumbv.  Vector  CSP,  LLC 

■  “Generic  Sensor  Model”.  Dr.  Stanley  Hack.  Lockheed  Martin  MS-2 

■  “Event  Timeline  Analysis  in  Multi  Mission  Scenarios  with  System  Simulation  Models”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  6 

■  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina  Research  Authority  (SCRA) 

■  “Agile  Governance  for  SOA-Based  Military  Systems  of  Systems”,  Mr.  Robert  Beck,  Villanova  University 

■  “Reducing  Acquisition  Costs  Through  Incremental  Upgrades  by  Migrating  to  SOA”,  Mr.  Tim  Greer,  Lockheed  Martin  Corporation 
Track  7 

■  “The  DoD’s  Proactive  Approach  to  Emerging  Contaminants:  Managing  Risks  Today  for  Tomorrow’s  Warfighter  and  Mission  Readiness”,  Dr. 
Carole  LeBlanc,  Office  of  the  Deputy  Under  Secretary  of  Defense 

■  “ Safe-Escape  Analysis  System  Sa  fety  Engineering  Study”,  (paper!  (slide)  Mr.  David  Hall,  SURVICE  Engineering  Company 

Track  8 


■  “Development  of  Systems  Engineers  in  the  Sensors  &  SONAR  Systems  Department”.  Mr.  Lawrence  Lazar.Naval  Undersea  Warfare 
Center 

■  ^Systems  Engineering  and  the  Art  of  Seeing”,  Dr.  Robert  Monson,  Lockheed  Martin  Corporation 

■  “Understanding  Social  Networks- A  Key  Requirement  for  System  Engineers”,  Mr.  Karl  Selke,  Systems  Engineering  Analyst,  Evidence 
Based  Research,  Inc. 
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Conference  Agenda  {At  A  Glance} 

Sunday,  October  21, 2007 

5:00  pm  -  7:00  pm  Registration  for  Tutorials  and  General  Conference 

(Tutorials  are  an  additional  $225.00  registration  fee) 

Monday,  October  22, 2007 


7:00  am  -  6:00  pm 
7:00  am  -  8:00  am 

8:00  am  - 11:45  am 

12:00  pm  - 1:00  pm 
1:00  pm  -  5:00  pm 
5:00  pm  -  6:00  pm 


Registration 

Continental  Breakfast  for  Tutorial  Attendees  ONLY 
(Tutorials  are  an  additional  $225.00  registration  fee) 

Tutorial  Tracks 

(Please  refer  to  pages  4-5  for  Tutorial  schedule) 

Lunch  for  Tutorial  Attendees  ONLY 
Tutorial  Tracks  Continued 

Reception  in  the  Regatta  Pavilion  (Open  to  All  Participants) 


Tuesday,  October  23, 2007 


7:15  am -6:30  pm 
7:15  am  -  8:15  am 


Registration 
Continental  Breakfast 


8:15  am  -  8:30  am 


8:30  am  -  9:45  am 


9:45  am  - 10:00  am 


Introductions  8C  Opening  Remarks: 

Mr.  Sam  Campagna,  Director ;  Operations ,  NDIA 
Mr.  Boh  Rassa,  Director ;  Systems  Supportability,  Raytheon , 

Chain  Systems  Engineering  Division,  NDIA 

Keynote  Addresses: 

HON James  Finley ,  Deputy  Under  Secretary  of  Defense,  Acquisiton  &  Technology 
HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation 


Break 


2  10th  Annual  Systems  Engineering  Conference  ♦  October  22-25,  2007  ♦  San  Diego,  CA 


10:00  am  - 12:00  pm 

Plenary  Session:  Executive  Panel 

Moderator: 

Mn  Mark  Schaeffer ;  Director ;  Office  of  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics;  Director ;  Systems  and  Software  Engineering 

Panelists: 

Mn  Terry  Jaggers,  SAF/AQR  -  Science,  Technology,  and  Engineering 

Mn  Carl  Siel,  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy 
Research,  Development  and  Acquisition 

Mn  Doug  Wiltsie,  HQDA ,  OASA  (ALT) 

12:00  pm  - 1:30  pm 

Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Mn  Mike  Kern,  Senior  Systems  Engineer,  OASD  (Nil) 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

5:00  pm  -  6:30  pm 

Reception  in  the  Regatta  Pavilion 

Wednesday,  October  24, 2007 

7:00  am  -  5:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:30  pm 

Awards  Luncheon  in  the  Regatta  Pavilion 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

Thursday,  October  25, 2007 

7:00  am  -  3:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:00  pm 

Luncheon  in  the  Regatta  Pavilion 

1:00  pm  -  3:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

3:00  pm 

Conference  Adjourns 

7:00  am  Registration  &  Continental  Breakfast 
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Tutorial  Sessions  -  Monday,  October  22, 2007 

8:00  am  -  9:45  am  10:15  am  - 1 1 :45  am 


Track  1 

Tutorial 

Session  1A1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1A2 

5454  -  Cost  As  an  Independent  Variable 
and  Trade  Studies 

Mr,  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1A3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices 

Mr.  Gary  Langford,  The  Naval  Postgraduate 

School 

Track  4 

Tutorial 

Session  1A4 

5498  -  System  Verification  Organization 

Mr,  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1A5 

Track  6 

Tutorial 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention 

Session  1A6 

Mr.  Timothy  Olson,  Lean  Solutions  Institute, 
Inc. 

Track  7 

Tutorial 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  Users  Needs 

Session  1A7 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1A8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM) 

Mr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1B1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1B2 

5454  -  Cost  As  an  Independent  Variable  and 
Trade  Studies  (Contd) 

Mr.  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1B3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate 
School 

Track  4 

Tutorial 

Session  1B4 

5498  -  System  Verification  Organization 
(Cont'd) 

Mr.  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1B5 

Track  6 

Tutorial 

Session  1B6 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention  (Cont'd) 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1B7 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  User's  Needs 
(Cont'd) 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1B8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM)  (Cont'd) 

Mr.  Ahe  Meilich,  Lockheed  Martin 
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Buffet  Lunch 


3:15  pm  -  5:00  pm 


1 :00  pm  -  2:45  pm 


Track  1 

Tutorial 

Session  1C1 

5307  -  Requirements  Development  and 
Management 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1C2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1C3 

5404  -  How  to  Reduce  Schedule  Uncertainty 
by  Integrating  Sound  Management  Methods 
with  Systems  Engineering  Best  Practices 
(Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1C4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1C5 

5540  -  Introduction  to  Reliability  Analysis 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1C6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1C7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1C8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1D1 

5307  -  Requirements  Development  and 
Management  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1D2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management  (Cont'd) 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1D3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1D4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 
(Cont'd) 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1D5 

5540  -  Introduction  to  Reliability  Analysis 
(Cont'd) 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1D6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc . 

Track  7 

Tutorial 

Session  1D7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews  (Cont'd) 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1D8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM)  (Cont'd) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

5:00  pm  -  6:00  pm  Reception  in  Regatta  Pavilion 


Tuesday,  October  23,  2007 


5:00  pm  -  6:00  pm  Reception  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 


12:00  pm  Lunch  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 
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Thursday,  October  25,  2007 

8:00  am  -  9:45  am  concurrent  sessions  10:15  am  - 12:00  pm 
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12:00  pm  Lunch  in  the  Regatta  Pavilion 


Thursday,  October  25,  2007 


concurrent  sessions 

:30  pm  -  3:00  pm 

Bayview 

A 

TRACK  2 

Systems  Engineering 
Effectiveness 

Session  4C2 

5814  -  Defense  System  and 
Large-Scale  Systems  Based  on 
Terminal  Control 

Mr.  Xiang-Wen  Xiong,  Zhongheng 
High-Tech  Institute,  Inc. 

Bayview 

B 

TRACK  4 

Logistics,  Supportability, 
and  Readiness 

Session  4C4 

5416  -  The  Deployment  Readiness 
Service:  The  Challenges  of 
Implementing  a  Service  Oriented 
Architecture  in  a  Legacy  System 
Environment 

Mr.  George  Dalton,  USAF 

yview 

C 

TRACK  5 

Modeling  & 

Simulation 

5838  -  Generic  Sensor  Model 

5852  -  Event  Timeline  Analysis 
in  Multi  Mission  Scenarios  with 
System  Simulation  Models 

5428  -  A  Prototype  Tool  for 
Concept  Design  Modeling  and 
Optimization  of  Combat  Systems 

CQ 

Session  4C5 

Dr.  Stanley  Hack,  Lockheed 

Martin  MS-2 

Mr.  Ravi  Moorthy,  Lockheed 
Martin  MS2 

Mr.  Vikram  Ganesan,  General 
Dynamics  Land  Systems 

Conference  Adjourns 


Systems  Engineering  Effectiveness: 

Mr.  A1  Brown 
Ms.  Sharon  Vannucci 
Mr.  Bob  Lyons 

Logistics  Supportability  &  Readiness: 

Mr.  Chuck  Silva 
Mr.  Joel  Moorvich 

Test  &  Evaluation  in  Systems  Engineering: 
Col  Rich  Stuckey,  USAF 
Mr.  Tom  Wissink 
Program  Management: 

Mr.  Hal  Wilson 
Modeling  &  Simulation: 

Mr.  Jim  Hollenbach 
Mr.  Gary  Belie 
Net  Centric  Operations: 

Mr.  Jack  Zavin 
Dr.  Rich  Eilers 
Dr.  Tom  Wickstrom 
Best  Practices  &  Standardization: 

Mr.  Paul  Croll 
Software: 

Dr.  Tom  Christian 
Education  &  Training: 

Mr.  George  Mooney 
Integrated  Diognostics: 

Mr.  Howard  Savage 
Mr.  Dennis  Hecht 
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Are  We  Ready  for  CMMI®.P  If  Not,  Lets  Fix  Ourselves 

Mr*  A1  Florence 

1A2 

5454 

Cost  As  an  Independent  Variable  and  Trade  Studies 

Mr.  Ed  Casey 

1A3 

5404 

How  to  Reduce  Schedule  Uncertainty  by  Integrating  Sound  Management 
Methods  with  Systems  Engineering  Best  Practices 

Mr.  Gary  Langford 

1A4 
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System  Verification  Organization  Tutorial 

Mr.  Jeffrey  Grady 

1A6 

5542 

Best-In- Class  Early  Defect  Detection  and  Defect  Prevention 

Mr.  Timothy  Olson 

1A7 
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Introduction  to  SysML  &  Object  Oriented  Systems  Engineering 
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Mr.  Abe  Meilich 

1C1 
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5511 

Modeling  Sustainment  and  Risk  Mitigation  for  Net- Enabled  Realities 

Mr.  Philip  Boxer 
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5378 
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Mr.  Robert  Skalamera 
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Mr.  Dennis  Goldenson 
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5402 
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Mr.  Chris  DiPetto 
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5531 
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Mr.  Samuel  Son 
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5508 

Integrated  Structural  Health  Monitoring  System  Using  Lamb  Waves 

Maj  Joerg  Walter,  US AF 

Dr.  Som  Soni 

2C4 

5797 

Environment,  Safety,  and  Occupational  Health  (ESOH)  -  Design 
Considerations  to  Support  Sustainability  and  Readiness 

Ms.  Patricia  Huheey 

Ms.  Karen  Gill 

2C5 

5421 

M&S -Related  Guidelines  for  Contracting 

Mr.  Michael  Truelove 

2C5 

5446 

Implementing  the  Acquisition  M&S  Master  Plan 

Mr.  Michael  Truelove 

Mr.  Jim  Hollenbach 

2C5 

5606 

M&S  Planning  and  Employment  Best  Practices 

Mr.  Jim  Hollenbach 

2C6 

5430 

The  Use  of  Enterprise  Architecture  to  Support  the  Development  of  the  Next 
Generation  Air  Transportation  System  (NextGen) 

Mr.  Jerry  Friedman 

Mr.  David  Snyder 

2C6 

5500 

Complex  Systems  of  Systems:  The  Dual  Challenge 

Mr.  Philip  Boxer 

Ms.  Lisa  Brownsword 

Mr.  Ed  Morris 

2C7 

5491 

Systems  Engineering  Process  Improvements  at  NAVAIR 

Mr.  Michael  Gaydar 

2C7 

5780 

Simplifying  &  Scaling  Engineering  Processes:  Unifying  Business  Units  and 
Engineering  Disciplines 

Mr.  Raymond  Jorgensen 

2D1 

5405 

Systems  Engineering  Realization  for  the  Future 

Ms.  Karen  Bausman 

2D1 

5484 

Tools  and  Resources  to  Enable  Systems  Engineering  Improvement 

Mr.  Michael  Kutch,  Jr. 

Mr.  Michael  Knox 

2D1 

5626 

Good  Systems  Engineering  Insures  Good  Producibility 

Dr*  Thomas  Christian,  Jr* 

Mr*  Rich  Stepler 

Mr*  Hamid  Akhbar 

2D2 

5489 

Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate 
Evaluation 

Mr*  Joseph  Tribble 

Mr*  Matthew  Rodakis 

2D2 

5774 

Implementing  and  Measuring  Test  Program  in  a  Sustainment  Environment 

Mr*  James  Miller 

2D3 

5365 

PValue  of  Systems  Engineering:  Analysis  &  Results  from  Previous  and 

Current  Studies  of  Over  100  System  Development  Projects 

Mr.  Allan  Mink,  II 

2D3 

5435 

Reforming  the  DoD  Acquisition  Process  -  A  Systems  Engineering  Approach 

Mr*  Stephen  Ward 

Mr*  Christopher  Perkins 

2D3 

5779 

Applying  Systems  Engineering  During  Pre-Acquisition  Activities 

Lt  Col  Mark  Wilson,  US  AF 

Mr.  Jeff  Loren 

2D4 

5271 

Leveraging  EMS  for  Condition  Based  Maintenance 

Mr*  Thomas  Hawley 

2D4 

5287 

Sparing  Satellites— Comparitive  Strategies  of  In-Orbit  &  In-Factory  Storage 

Mr*  James  Mazzei 

Mr  James  Ayers 

Ms*  Camille  Keeley 

Mr*  Jon  Westergaard 

2D4 

5368 

Real-Time  Diagnostics  for  High  Availability  Systems 

Mr*  Edward  Beck 

2D5 

5427 

Modeling  and  Simulation  Resource  Reuse  Business  Model 

Mr*  Dennis  Shea 

Dr*  John  Hampson 

2D5 

5603 

Modeling  and  Simulation  Support  Plan 

Mr*  David  Henry 

2D5 

6099 

Modeling  and  Simulation  Education  for  the  Acquisition  Workforce: 
Requirements  Analysis 

Mr*  David  Olwell 

Ms*  Jean  Johnson 

2D6 

5576 

Systems  Engineering  in  the  Cognitive  and  Social  Domains  of  Net  Centric 
Operations 

Dr*  Abe  Meilich 

2D6 

Global  Information  Grid  (GIG),  Technical  Foundation  (GTF)  and  GIG 
Compliance  Assessment  (GICA) 

Mr*  Brendan  Goode 

2D6 

Global  Information  Grid  (GIG)  Performance  Assessment  Framework 

Mr*  Tony  Modelfino 

2D7 

5781 

Integrating  Engineering  Project  Management  and  Product  Development 
Processes 

Mr*  Raymond  Jorgensen 

2D7 

5831 

Engineering  for  System  Assurance  -  Legacy  Life  Cycle,  Leadership 

Mr*  Paul  Croll 

2D8 

5315 

Requirements  for  a  Chief  Software  Engineering  in  a  DoD  Acquisition  Agency 

Mr*  A1  Florence 

2D8 

5481 

Developing  An  Integrated  Process  Methodology  For  Interim  Software 

Releases 

Mr*  Tim  Woods 

Mr*  Jerrell  Stracener 

3A1 

5536 

A  Day  in  the  Life  of  a  Verification  Requirement 

Mr*  Stephen  Scukanec 

Mr*  James  van  Gaasbeek 

3A1 

5806 

Investigating  the  Use  of  SysML  on  the  FBX-T  Radar  Program 

Mr*  Chad  Schuyler 

Mr*  MarkMinnucci 

Ms*  Caroline  Elias 

3A1 

5824 

Change  Management  of  UML-Based  Systems  Engineering  Artefacts 

Mr*  David  Price 

3A2 

5520 

System  Test  and  Evaluation  in  the  DARPA  Immune  Building  Demonstration 
Program 

Mr*  Mark  Saxon 

Mr*  James  Risser 

3A2 

5553 

Modeling  and  Simulation  in  the  Navy  Warfare  Systems  Test  &  Evaluation 
Enterprise 

Ms*  Shala  Malone 

Mr*  Richard  Reading 

3A3 

5523 

Systems  Engineering  Plan  Preparation  Guide  Update 

Mr*  Chester  Bracuto 

3A3 

5525 

Toward  a  Unified  Systems  Engineering  Plan 

Mr*  Robert  Scheurer 

3A3 

5587 

Integrating  Risk  &  Knowledge  Management 

Mr*  David  Lengyel 

3A4 

5546 

Defining  Lean  Service  and  Maintenance  Processes 

Mr*  Timothy  Olson 

3A5 

5538 

Advancing  the  FEDEP  for  Simulation  Based  Acquisition 

Dr*  Katherine  Morse 

Mr*  Paul  Lowe 

3A5 

5614 

Live  Virtual  Constructive  (LVC)  Architecture  Interoperability  Assessment 

Mr*  Warren  Bizub 

Mr*  Ken  Goad 

3A6 

5497 

Achieving  Agility  in  Cyberspace 

Mr*  Philip  Boxer 

Mr,  Edwin  Morris 

3A6 

5815 

Distributed  Firewalls 

Mr,  Alejandro  Gastelum 

3A7 

5592 

CMMI — Next  Steps 

Ms.  Kristen  Baldwin 

Mr.  Lawrence  Osiecki 

Dr.  Karen  Richter 

3A7 

5743 

CMMI  Instructional  Challenges  to  Systems  Engineers  in  Small  Settings 

Dr.  Mary  Anne  Herndon 

3A8 

5526 

Defining  Software  Component  Specifications:  An  Open  Approach 

Mr.  Kenneth  Klein 

3A8 

5676 

System  Engineering  and  Software  Exception  Handling  (SEH) 

Mr.  Herbert  Hecht 

3B1 

5557 

How  to  Measurably  Improve  Your  Requirements 

Mr.  Timothy  Olson 

3B1 

5670 

Review  of  the  Roles  of  a  System  Architect 

Mr.  Tomasz  Lomecki 

Dr.  Dinesh  Verma 

Mr.  Eirik  Hole 

3B1 

5865 

Case  Studies:  A  Common  Language  Between  Engineers  and  Managers 

Capt  DeWitt  Latimer  IV,  USAF 

3B2 

5473 

Joint  Mission  Environment  Test  Capability  ( JMETC) 

Mr.  Richard  Lockhart 

3B2 

5585 

Testing  Concept  of  Operations  in  DoD's  Net  Centric  Environment 

Mr.  John  Eleazer 

Mr.  Charles  Reeder 

3B3 

5381 

Systems  Engineering  and  Program  Management  -  How  Different  are  They? 

Ms.  Lori  Zipes 

3B3 

5799 

Systems  Engineering  Analysis  to  Improve  Concept  Development  of  Complex 
Defense  Systems 

Mr.  Michael  Harper 

Mr.  Philip  Butler 

Dr.  Jerrell  Stracener 

3B4 

5530 

Considerations  &  Propose  Approach  for  Integrating  New  Hardware  and 
Software  into  the  Legacy  Military  Aircraft  Avionics  Systems  -  a  Systems 
Engineering  Lesson-Learned  Perspective  on  the  C-17  Program 

Mr.  Phat  Vu 

3B4 

5903 

Asset-Based  PBL  for  Navy  Warships  -  A  Case  Study  for  LCS  Class  Ship 

Mr.  Michael  Mahon 

3B4 

5954 

Integrated  Diagnostics  Closed  Loop  Data  System  (At  the  Point  of  Use) 
(Support  Systems  Knowledge  Engineering  Enhances  Traditional  Support 
Equipment  Systems  Engineering) 

Mr.  Steven  Head 

Mr.  John  Bauer 

3B5 

5426 

A  Methodology  for  Assessing  and  Prioritizing  the  Risks  Associated  with  the 
Level  of  Verification,  Validation  and  Accreditation  (VV&A)  of  Models  and 
Simulations 

Dr.  James  Elele 

3B5 

5431 

Standardized  Documentation  for  Verification,  Validation,  and  Accreditation 
-  A  Status  Report  to  the  Systems  Engineering  Community 

Mr.  Kevin  Charlow 

Mr.  Curtis  Blais 

Mr.  David  Broyles 

Ms.  Marcy  Stutzman 

3B6 

5386 

Application  of  Autonomic  Agents  for  Global  Information  Grid 

Mr.  David  Cox 

Mr.  Youssif  Al-Nashif 

Dr.  Salim  Hariri 

3B6 

5847 

Architecture-Based  Concept  Evaluation  in  Support  of  JCIDS 

Dr.  David  Jacques 

Dr.  John  Colombi 

3B7 

5485 

Executing  a  Successful  CMMI  Maturity  Level  3  SCAMPI  For  SPAWAR 
Systems  Center  Charleston 

Mr.  Michael  Kutch 

Ms.  Sandra  Guidry 

3B7 

5808 

FISMA  Operational  Controls  and  Their  Relationship  to  Process  Maturity 

Mr.  Ronda  Henning 

3B8 

5764 

A  Convergence  of  Technologies  for  Better  Software  NOW! 

Mr.  Dorothy  Acton 

3B8 

5803 

Identifying  Acquisition  Patterns  of  Failure  Using  System  Archetypes 

Mr.  William  Novak 

Dr.  Linda  Levine 

3B8 

5883 

Acoustic  Rapid  COTS  Insertion  (ARCI)  Advanced  Processor  Build  (APB) 
Systems 

Mr.  Gary  Tissandier 

3C1 

5594 

Handbook  on  Engineering  for  System  Assurance 

Ms.  Kristen  Baldwin 

Ms.  Christine  Hines 

3C1 

5805 

A  Pragmatic  Approach  for  Defining  and  Utilizing  System  States  and  Modes 

Mr.  Mark  Minnucci 

Mr.  Chad  Schuyler 

Ms.  Caroline  Elias 

3C2 

5380 

Applications  of  Systems  Engineering  to  Pre-Milestone  A  Projects 

Ms*  Lori  Zipes 

3C2 

5464 

“Do  it  right,  do  it  early;  Do  it  early  Do  it  right”  —  Considerations  for  the 

Early  Stages  of  Concept,  System,  and  Systems- of-Systems  Definition 

Mr*  Jeff  Loren 

LtCol  Mark  Wilson 

3C3 

5450 

Airborne  Signals  Intelligence  Payload  (ASIP)  Program  Integrated  Risk 
Management 

Mr*  Daniel  Bolek 

3C3 

5795 

Lockheed  Martin  Aeronautics  Company  Approach  to  Solving  Systemic 
Development  Program  Issues 

Dr*  John  Weaver 

3C4 

5477 

Ml 09  Howitzer  Sustainment 

Mr*  Peter  Henry 

Mr*  Daniel  Malinowski 

Mr*  Manohar  (Manu)  Maman 

3C4 

5843 

Aging  Aircraft  Sustainment  with  Non-Standard  Engineering 

Ms*  Kendal  Hinton 

Chris  Fowler 

3C5 

5376 

Experiences  in  Applying  SysML  to  Develop  Interoperable  Torpedo  Modeling 
and  Simulation  Components 

Mr*  Thomas  Haley 

3C5 

5740 

Leveraging  the  Integrated  Engineering  Model  (IEM)  Process  as  a  Lead 

Systems  Integrator 

Mr*  Anthony  Montano 

3C6 

5519 

Case  Study:  Net  Centric  Mission  Thread  Modeling  and  Analysis 

Dr*  Premjain 

Mr*  Brian  Pridemore 

3C6 

5848 

System  of  Systems  Implications  for  Operational  Test 

Dr*  John  Colombi 

Dr*  David  Jacques 

3C7 

5760 

How  to  Paint  a  Room:  The  Role  of  Specs  &  Standards  in  SE 

Mr*  Robert  Kuhnen 

3C7 

5837 

Continuous  Improvement  at  the  Organization,  Team,  and  Individual  Levels 
-  Lessons  Learned  Integrating  CMM,  TSP,  and  PSP  and  Why  All  Three  are 
Needed 

Mr*  Girish  Seshagiri 

3C8 

5350 

Revitalizing  Education  and  Training  in  Systems  Engineering 

Dr*  Don  Gelosh 

3C8 

5501 

Customer- Driven,  Partnership -Based  Systems  Engineering  Education  and 
Training 

Dr*  Jerrell  Stracener 

Dr*  Steven  Szygenda 

Mr*  James  Rodenkirch 

3D1 

5433 

A  Practical  Application  of  Structured  System  Engineering  and  Failure  Mode 
Effects  Analysis  to  New  Technologies 

Mr*  Paul  Deniston 

3D1 

5505 

Discussion  of  the  LIS*  Army  RDECOM  APS  Objective  Trade  Study 

Mr*  Frank  Salvatore 

3D2 

5593 

Systems  Engineering  in  a  Systems  of  Systems  Environment  -  Defense  Update 

Ms*  Kristen  Baldwin 

Dr*  Judith  Dahmann 

Mr*  Ralph  Lowry 

3D2 

5602 

ASW  System- of-Systems  Engineering  Analysis  Applied  in  an  LCS  ASW 
Integration  Pilot  Project 

Mr*  G*  Richard  Thompson 

3D2 

5832 

A  Brigade  Capability  Approach  to  the  Evolution  of  Current  Ground  Combat 
Systems 

Ms*  Roberta  Desmond 

Mr*  Kenneth  Mick 

Mr*  Rick  Burtnett 

3D3 

5388 

Integrated  Risk  and  Earned  Value  Management 

Mr*  Paul  Davis 

3D3 

5434 

Improvements  to  the  Risk  Management  Process 

Mr*  Doug  Atkinson 

Ms*  Amy  Mercado  Vince 

3D3 

5800 

Application  of  Risk  Management  Practices  to  NNSA  Tritium  Readiness 
Subprogram 

Mr*  Sham  Shete 

Mr*  Srini  Venkatesh 

3D4 

5528 

C-17  Program  Applies  Systems  Engineering  to  a  Large  Improvement  Project 

Mr*  Brent  Theodore 

3D4 

5854 

Effective  Time  on  Station  (ETOS):  The  Persistent  Performance  Metric  for 
Aircraft  Systems 

Mr*  Christopher  Marchefsky 

3D5 

5409 

Information  Modeling  for  Systems  Integration 

Ms*  Claudia  Rose 

Mr*  A1  Brenner 

Ms*  Kim  Idol 

3D5 

5575 

Simulation  Supported  Decision  Making 

Mr*  Gene  Allen 

3D5 

5835 

Unifying  Systems  Engineering  Simulations 

Mr*  Kevin  Tang 

Mr*  Glenn  Beach 

Mr*  Rakesh  Patel 

Mr*  Jason  Ueda 

3D6 

5407 

Quantitative  Comparison  of  Alternative  Designs  for  JC3M  System 

Mr*  Gregory  Miller 

Mr*  Ian  Finn 

3D6 

5445 

Advanced  Net  Centric  Simulation  for  Aerospace  Command  and  Control 

Ms*  Kimberly  Kendall 

3D7 

Anatomy  of  an  Award  Winning  Safety  Program:  A  Case  Study  of  the  SSGN 
OHIO  Class  Conversion  Safety  Program 

Mr*  Ricky  Milnarik 

Mr*  Bryan  Stanley 

Mr*  Thomas  Cook 

Mr*  Norman  Gauthier 

3D7 

Addressing  Environment,  Safety  and  Occupational  Health  Issues  for  the 

Mine  Resistant  Ambush  Protected  (MRAP)  Vehicle  Program 

Ms*  Jennifer  Malone 

3D7 

Mr*  Mike  Demmick 

3D8 

5438 

Autopsy  of  A  Good  Systems  Engineer  -  An  Endangered  Species 

Mr*  Jimmy  Thai 

3D8 

5522 

Systemic  Root  Cause  Analysis  of  Acquisition  Program  Issues 

Mr*  Dave  Castellano 

Ms*  Laura  Dwinnell 

3D8 

5873 

Reference  Curriculum  for  a  Graduate  Program  in  Systems  Engineering 

Dr*  Rashmi  Jain 

Dr*  Dinesh  Verma 

Mrs*  Anithashree  Chandrasekara 

4A1 

5499 

A  Robust  Process  for  Resolving  Interface  Design  Issues  in  the  Complex 
Concurrent  LCS  System  Engineering  Environment 

Mr*  William  Traganza 

Mr*  Joseph  Conway 

Mr*  Joseph  Darwood 

4A1 

5785 

UAI  Part  1:  Increasing  Flexibility  Through  Data  Driven  Interfaces;  Part  2: 
Describing  Flexibility  as  an  Operational  Capability 

Mr*  Jonathan  Miller 

Mr*  Oren  Edwards 

4A2 

5443 

Technology  Readiness  Assessments;  Milestone  B  Certification  Requirement 
for  Technologies  To  Be  Demonstrated  in  a  Relevant  Environment 

Dr*  Jay  Mandelbaum 

4A2 

5453 

Meeting  Enterprise  System  Engineering  Challenges  for  the  US*  Next 
Generation  Air  Transportation  System  (NextGen) 

Ms*  Catherine  Bolczak 

Mr*  Robert  Humbertson 

Mr*  John  Mack 

Mr*  Gerald  Friedman 

4A2 

5675 

Implementing  the  Technology  Maturity  Vector 

Mr*  Joseph  Terlizzese,  Jr* 

4A3 

5532 

Systems  Engineering  Approach  to  Prioritize  Projects 

Mr*  Thomas  Condron 

Maj  Christopher  Lardner 

4A3 

5849 

Managing  Requirements  to  Manage  Scope  in  the  Case  of  MUOS 

Ms*  Christy  Howard 

Ms*  Debra  Shannon 

CDR  Ralph  (Trip)  Braund 

4A3 

5867 

Organizational  Leadership  and  Management  Dynamics  for  Technical 
Execution  in  Acquisition  Programs 

Mr*  Francis  Sisti 

Captain  DeWitt  Latimer,  IV 

4A4 

5419 

DoD/OSD  Sustainment/ Readiness  Initiatives:  Reduction  of  Total 

Ownership  Costs  (R-TOC)  and  Value  Engineering  ( VE) 

Dr*  Danny  Reed 

Mr*  David  Erickson 

4A4 

5493 

Innovation  Strategies  for  Affordable  Readiness 

Mr*  Tom  Choinski 

4A4 

5629 

Relating  Investment  In  Reliability  Engineering  To  Reliability  Improvement 
And  Reduction  Of  Total  Ownership  Cost 

Dr*  James  Forbes 

Mr*  E*  Andrew  Long 

4A5 

5444 

Aircraft  Flight  Simulator  Acceptance  Criteria 

Mr*  Dean  Carico 

4A5 

5829 

Computer  Modeling  to  Solve  Problems  with  the  T-38  Propulsion 
Modernization  Program 

Mr*  Randall  Wimer 

Mr*  Andrew  Hall 

4A6 

5798 

Agile  Governance  for  SOA-Based  Military  Systems  of  Systems 

Dr*  Robert  Beck 

Ms*  Jessica  Byrnes 

Ms*  Elliot  Sloane 

Ms*  Sue  Metzger 

4A7 

5377 

Risk  and  System  Safety  in  Aerospace  Systems  Engineering 

Dr*  Daniel  Schrage 

4A7 

5439 

Safe- Escape  Analysis  System  Safety  Engineering  Study 

Ms*  David  Hall 

Mr*  Kenneth  Chirkis 

4A7 

5460 

The  DoD’s  Proactive  Approach  to  Emerging  Contaminants:  Managing  Risks 
Today  for  Tomorrow's  Warfighter  and  Mission  Readiness 

Dr*  Carole  LeBlanc 

4A8 

5494 

Development  of  Systems  Engineers  in  the  Sensors  &  SONAR  Systems 
Department 

Mr*  Lawrence  Lazar 

4A8 

5617 

Systems  Engineering  and  the  Art  of  Seeing 

Dr*  Robert  Monson 

4A8 

5792 

Understanding  Social  Networks-  A  Key  Requirement  for  System  Engineers 

Mr.  Karl  Selke 

4B1 

5490 

USAF  Type  Certification  of  Commercial  Derivative  Aircraft 

Mr*  Thomas  Morgan 

Mr*  Joel  Ligon 

4B1 

6135 

Global  Positioning  System  Case  Study 

Mr*  Randall  Bullard 

4B2 

5369 

Benchmarking  Operational  Configuration  in  a  Spiral  Development  Process: 
Ballistic  Missile  Defense  System 

Mr*  Hannibal  Wright 

4B2 

5759 

Sensor  Resource  Allocation  as  a  Driver  in  Ballistic  Missile  Defense  System 
Concept  Development 

Mr*  Ravi  Moorthy 

Mr*  Jay  Davidson 

Mr*  Mark  Russel 

4B3 

5850 

How  to  Talk  to  a  Program  Manager 

Ms*  Anita  Carleton 

Dr*  William  Nichols 

Dr*  John  Mishler 

4B4 

5773 

Implementing  a  Systems  Engineering  Risk  Program  in  a  Sustainment 
Environment 

Mr*  James  Miller 

4B4 

5826 

Risk  Management  and  Software  Life  Cycle  Selection 

Mr*  Thomas  McDermott 

4B5 

5355 

Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the 

US*  Coast  Guard 

Mr*  Patrick  Cumby 

4B5 

5397 

Efficacy  of  Modeling  &  Simulation  in  Defense  Life  Cycle  Engineering 

Mr*  Donald  Cox 

Dr*  Salim  Hariri 

4B6 

5320 

Toward  an  Intelligent  Military  Enterprise:  An  Introduction  to  an  Approach 
to  Joint  Forces  Based  on  Meta-Systems  Theory 

Dr*  Kent  Palmer 

4B6 

5820 

Reducing  Acquisition  Costs  Through  Incremental  Upgrades  by  Migrating  to 
SOA 

Mr*  Tim  Greer 

4B7 

Overview  of  DoD  Environment,  Safety  and  Occupational  Health 
Requirements,  Terminology,  System  Safety  Methodology  and  Risk 

Assessment 

Mr*  Sherman  Forbes 

4B7 

Overview  of  Environment,  Safety  and  Occupational  Health  Activities  Within 
Systems  Engineering 

Mr*  Sherman  Forbes 

4C2 

5814 

Defense  System  and  Large-Scale  Systems  Based  on  Terminal  Control 

Mr*  Xiang- Wen  Xiong 

4C4 

5416 

The  Deployment  Readiness  Service:  A  Case  Study  of  the  Challenges 
of  Implementing  a  Service  Oriented  Architecture  in  a  Legacy  System 
Environment 

Mr*  George  Dalton,  II 

Dr*  Robert  Mills 

4C5 

5428 

A  Prototype  Tool  for  Concept  Design  Modeling  and  Optimization  of 

Combat  Systems 

Mr*  Vikram  Ganesan 

Mr*  Philip  Morgan 

4C5 

5838 

Generic  Sensor  Model 

Dr*  Stanley  Hack 

Dr*  Josef  Keith 

4C5 

5852 

Event  Time  Line  Analysis  in  Multi  Mission  Scenarios  with  System 

Simulation  Models 

Mr*  Ravi  Moorthy 

Mr*  Paolo  Trinchieri 

Mr*  Todd  Brown 

Displayers 

California  Institute  of  Technology 
Industrial  Relations  Center 

Defense  Acquisition  University 

Defense  Acquisition  University,  West  Region 

Georgia  Tech  Research  Institute 

Headquarters  Air  Force  Material  Command 

INCOSE 

Lean  Solutions  Institute,  Inc* 

Project  Performance  International 
The  MITRE  Corporation 
TYX  Corporation 
University  of  Missouri-Rolla 


Vitech  Corporation 


Notes 


a 


o 


vector/CSP 


Transforming  USCG  Logistics 


A  systems  engineering  approach  to  transforming  the  USCG  Enterprise 
Logistics  Systems 


Patrick  W.  Cumby 

Director  Performance  Systems 

VectorCSP 


www.  vectorcsp.  com 


vector/CSP 


Agenda 


•  USCG  Logistics  T ransformation 
Background 

•  Enterprise  T ransformation  Basics 

•  USCG  Logistics  T ransformation 

•  Demos  of  Logistics  Models  and 
Transition  Dashboards 
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Logistics  Transformation  Background 


•  USCG  has  several  "stovepiped"  logistics 
business  models  (surface,  air,  shore,  IT) 

•  Models  have  evolved  over  time  and  are  not 
integrated  or  strategically  aligned 

•  Some  of  the  various  models  utilize  modern 
logistics  concepts,  others  do  not 

•  Limited  visibility  into  systems  performance 

•  Limited  ability  to  manage  costs  and 
effectiveness 

•  Then  along  comes  the  Deepwater 
program,  the  CFO  Act,  and  Katrina... 
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Logistics  Transformation  Drivers 


•  Deepwater  program  -  recapitalization 
of  USCG  assets  and  capabilities 

-  Deepwater  program  has  experienced 
several  issues  that  have  led  to  a  major 
restructuring  of  the  program. 

•  CFO  Act  -  Mandate  to  institute  total 
asset  visibility  and  financial  controls 

•  Success  of  Katrina  disaster  response  - 
demonstrated  strengths  of  USCG 
Aviation  logistics  model 
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Logistics  Transformation  Objectives/  Scope 


Admiral  Allen  issues  Cl  AO  #4 

Bi-level  maintenance  w/more  standardized 
procedures. 

Centralized  supply  chain  management  w/spending 
driven  by  maintenance  requirements. 

Disciplined/ standard  Coast  Guard-wide  engineering 
and  logistics  business  processes,  modeled  after  our 
internal  best  practices  currently  in  use  in  aviation. 

Strong  configuration  management  processes, 
w/associated  compliance  inspections,  to  ensure  all 
configurations  are  safe,  effective,  and  supportable 
when  installed. 

Reduce  the  number  of  financial  and  information 
systems. 
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Transformation  Support  Team 


•  Logistics  T ransformation  Program 
Integration  Office  (LTPIO)  established 

•  General  Dynamics  contracted  to 
provide  program  management 

•  VectorCSP  contracted  to  support 
organizational  and  logistics 
transformation 
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Transformation  Management  Approach 


•  LTPIO  chose  VectorCSP's  Pathfinder 
approach  to  develop  the  logistics 
business  model 

•  Pathfinder  is  a  systems-oriented, 
tools- based  transformation 
management  methodology 

•  Pathfinder  incorporates  a  complete 
business  systems  performance  model, 
with  an  emphasis  on  behavioral 
engineering 
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Let's  go  back  to  Org 
Transformation  Basics! 


Start 

Here 


Finish! 
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Enterprise  Transformation  ABCs 


A:  Where 

B:  Where 

you  are  now. 

you  want  to 
be. 

A 


C:  The  path 
from  A  to  B. 


B 
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A:  Where  you  are  now. 


•  I  n  order  to  get  from  A  to  B,  you 
have  to  understand  A. 

•  A  is  your  "As-ls"  organizational  and 
business  model 

•  A  is  usually  very,  very  complex... 


Usually  the  important 
cultural  and  political 
aspects  of  a  business 
model  are  not 
well  documented. 
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B:  Where  you  want  to  be. 


I  n  order  to  get  from  A  to  B,  you 
also  have  to  understand  B. 

B  is  your  'To-Be"  organizational  and 
business  model 

B  is  usually  not  well  defined... 
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C:  The  Path  from  A  to  B 


The  really, 
really,  really 
hard  part. 
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Speedbumps  on  the  Path 


Strategic  Alignment 

Technology 

Process 

Organizational  Design 
Culture 


(K 


s 


Question: 

Of  these 

roadblocks,  which 
is  most  difficult 
to  overcome? 


L 

1 

A 

'r  i 

•  Politics 

V=T=^ 

1  rr^ ' 

vector/CSP 


To  sum  it  up... 


You  move  from  a  highly 
complex  (and  usually 
broken)  system... 


...through  a  complex 
transformation  process 
fraught  with  cultural, 
technical,  and  political 
barriers... 


...to  get  to  what  is 
usually  a  poorly 
understood  end  state. 


2 


Question: 

What  percentage 
of  enterprise 
transformations 
succeed? 


±4 


I  t's  no  wonder  that 

85% 

of  all  enterprise  transformations 

Do  not  fully  succeed! 
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What  you  need  to  succeed 


1.  Clearly  defined  transformation  objectives 

2.  A  way  to  identify  A  and  B 

3.  A  roadmap  to  transform  the  structures, 
processes,  technology,  culture  and 
politics  of  the  organization 

4.  A  way  to  manage  the  astounding 
complexity  and  mountains  of  data  of 

such  a  large-scale  endeavor 

5.  A  way  to  communicate  with  all 
stakeholders 

6.  A  way  to  measure  success  of  the 

transformation 

7.  A  fully-dedicated  transformation 
management  team 
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I  n  other  words... 


...you  need  a  systems  engineering  approach! 


The  Coast  Guard  Logistics 
T  ransformat  on 


Start 

Here 


Finish! 
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Coast  Guard  Transformation 
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A:  Separate 
logistics  models 
for  surface  and 
aviation. 


C:  Logistics 
T  ransformati  on . 


B:  Standard 
logistics 
model  based 
on  aviation 
model. 


Path  from  A  to  B 


B 


Desired  State 
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A:  The  "As- 1  s"  State  of  CG  Logistics 


Multiple  logistics  models  (naval,  aviation, 
shore,  C4ISR) 

Problematic  logistics  systems  for  naval, 
shore,  and  technology  systems 

-  Non-standard  fleet  assets  and  inventories 

-  Antiquated  logistics  processes  and  technology 

-  Sub-optimal  acquisition  model 

-  Poor  financial  controls 

Not  compliant  with  CFO  Act 

Problems  with  Deepwater  program 

Getting  the  mission  accomplished  despite 
sub-optimal  logistics  systems  due  to 
dedication  and  "can-do"  attitude 
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B:  The  'To-Be"  State  of  CG  Logistics 


Adopt  CG  Aviation  I  ntegrated 
Logistics  Systems  (ILS)  model 

Standard  fleet  assets  and  Total  Asset 
Visibility 

I  ntegrated  technology  infrastructure 
(based  on  modified  ALMIS) 

Transparent  and  tightly  controlled 
financials 

CFO  Act  Compliance 

Systems  measures  of  effectiveness 
( MOEs) 


C:  The  Path  to  Transition 
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Commandant's  Cl  AO  establishes 
transformation  objectives 

LTPIO  established  as  the  fully- 
dedicated  transition  team 

Pathfinder  Performance 


Modeling  approach  chosen  by 
LTPI O  as  a  key  transformation 
management  tool. 
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About  Pathfinder 


vectoi^CS  P 

•  Pathfinder  is  a  dedicated 

transformation  modeling  and 

support  system 

•  Designed  for  large-scale 
organizational  transformations 

•  Pathfinder  is  based  upon  a  systems 
engineering  approach  to  org 
transformation 
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Performance  System  Engineering 


•  Pathfinder  breaks  organizational 
performance  into  discrete  systems  elements 

£  4^  y  -C  ft  . . . 

•  It  incorporates  process  engineering, 
organizational  design,  enterprise 
architecture,  and  most  importantly 
behavioral  engineering 

•  It  enables  modeling  of  all  elements  that 
influence  organizational  performance 

•  It  makes  it  possible  to  manage  the 
complexity  of  a  large  transformation 
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Performance  Systems  Components 


Key  building  blocks  of  a 
Performance  Model 

System 
System  Outcome 
J  ob  Role  or  Team  (People) 
Strategic  Objective 
4^  Policy 

p— ~ 

Technology 
-€!  Organization  ^ 


3 


Question: 

Which  of  these  elements 
is  most  critical  to  performance, 
yet  most  often  overlooked? 
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Performance  Model  Basics 
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A  performance  system  can 
have  multiple  subsystems 


System 

Outcome 


A  performance  system  is 
defined  by  its 
System  Outcomes 


The  Outcome  is  the 
central  element  of  a 
performance  model  26 


Examples  of  Systems  and  Outcomes 


•  The  Coast  Guard  logistics  performance  model  is 
based  on  the  standard  I LS  (I  ntegrated  Logistics 
System) 

•  Excerpt  from  the  CG  model: 


-j  ILS 


Maintenance 


Supply 


*4 


Cost  driver  components  identified 
Part  enrolled  in  ALMI S 
Level  of  Repair  determination 
Many  others... 
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Coast  Guard  I LS  System  Hierarchy 


Design  1  nterface 

Maintenance 

Manpower 

Supply 

Support  Equip. 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 
CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Tech  Data 

Training 

IT 

PHS&T 

Facilities 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 

CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 

1  nfo  Management 

Tech  data 

Envi  ronment/  Hazmat 
CM/ Standardization 
Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 
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Performance  Model  Relationships 


All  components 
of  the  model  are 
related  to 
Outcomes 


System 

Outcome 


What  job  roles  are  involved  in 
producing  the  outcome? 


What  policies  are  related 
to  the  outcome? 

What  strategic  objectives  does  the 
outcome  support? 


What  organizations  are  involved 
in  producing  the  outcome? 


What  technology  is  involved  in 
producing  the  outcome? 


What  systems  are  related  to 
the  outcome? 

J 


Modeling  A  and  B  for  CG  Logistics 


•  This  "performance  modeling" 
approach  is  used  to  create  as- is  (A) 
and  to-be  (B)  models  of  CG  logistics 


•  The  next  step  is  to  develop  the 
Logistics  Transformation  Mode!  (C) 
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Outcomes  as  Key  Transformation  Drivers 


'To-Be" 

System 

Outcome 


I  n  the  Pathfinder  transformation 
model,  system  outcomes  are  the  key 

transformation  drivers. 

in  other  words,  if  the  ' to-be  " 
outcome  can  be  achieved,  then  the 


transformation  to  that  part  of  the 
system  is  considered  a  success 


31 


vector/CSP 


Eliminating  Outcome  Performance  Gaps 


There  are  typically  gaps  between  an  as- is 
outcome  and  the  to-  be  outcome 

These  gaps  are  recognized  in  our  model  as 

a  performance  gap  element 

Some  gaps  are  simply  differences  in 
processes  or  personnel 

Other  gaps  may  require  technology, 
infrastructure,  or  organizational  changes  to 
eliminate 

These  gaps  must  be  eliminated  by  actions 

called  performance  interventions 
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Elements  of  the  Transformation  Framework 


vector/CSP 


•  Pathfinder  includes  gaps  and  interventions  as 
discrete  elements  of  the  transformation  model 


As-ls 

System 

Outcome 


To-Be 

System 

Outcome 


Performance 
I  intervention 


Performance 
I  ntervention 
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Coast  Guard  I  intervention  Examples 


•  Outcome:  Approved  Maintenance 
Procedure  Card  produced  I  AW 
COMDI NST  xxx.x 

-  Gap:  Maintenance  Requirements 
List  (MRL)  not  defined  for  surface 


•  I  ntervention:  Perform  MSG-3  logic  analysis 

•  I  ntervention:  Enroll  asset  in  ACMS 

•  I  ntervention:  Modify  MPC  process  guide 

•  I  ntervention:  Identify  and  train  MPC 
production  staff 
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About  I  ntewentions 
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Performance 
I  intervention 


Each  intervention: 

-  I  s  an  actionable  item 

-  Has  an  accountable  owner 

-  Has  schedule  constraints 

-  Has  associated  resources 

-  Can  be  used  to  build  a  transition  project 
plan 

-  And  most  importantly,  has  measurable 
success  criteria 
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Putting  the  elements  together... 


•  Strategies 

•  Systems 

•  Subsystems 

•  Outcomes 

•  I  nf I uencers  (policies,  organizations, 
people,  technology,  etc.) 

•  Gaps 

•  Interventions 

•  How  does  it  all  fit  together? 
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Transformation  Model  Framework 


V  4^ 


a 


* 


Strategic  Framework 

Systems  Framework  j 

Org  Framework 

Technology  Framework 

Org  Unit 
Performer: 

•Job  Role 

•  Team 

•  Automated  System 


IT  System 


IT  Feature 


Faci  I  ity/  Equi  pment 


Facility 


Tool/ Equi  pment 


Transition  Framework 


Transformation  Goal 


Performance  Gap 


Transformation  Objective 


I  ntervention 


T  ransition 
Success  Criteria 


The  USCG  Transition  Process 
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•  LTPI O  alignment  conducted,  transformation  objectives  identified 

•  Docs  and  resources  reviewed 

•  SMEs  identified 

•  I LS  systems  outcome- based  framework  developed 

•  Best  practices  identified  by  SMEs  (42) 

•  Preliminary  outcomes  identified  by  SMEs  (800) 

•  Org  and  strategic  models  defined 

•  Key  outcomes  identified  (250) 

•  Framework  relationships  to  key  outcomes  identified  by  SMEs  (docs,  job 
roles,  policies,  best  practices,  etc.) 

•  Skilled  Performers  (SPs)  identified  by  USCG 

•  SP  outcome  review  worksheets  prepared 

•  SP  outcomes  reviews  and  validation  conducted 

•  All  data  collected  in  Pathfinder  Performance  Modeler 

•  All  data  reviewed 

•  Analysis  reports  prepared 

•  PVs  and  Pis  developed  using  facilitated  meetings  with  LTPIO  working  groups 

•  Activity  crosswalks  prepared 

•  Project  plans  and  transition  dashboards  developed 

•  Transition  training  prepared  and  delivered 
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Aviation  Logistics  Framework  Scale 


800  aviation  logistics  model  outcomes  were 
identified  in  all  10  I LS  elements  (and  a 
cross-cutting  set  of  subelements) 

250  Key  Outcomes  (transformation  drivers) 
identified 

Key  Outcomes  mapped  to 

-  41  Systems 

-  698  J  ob  roles,  teams,  or  org  units 

-  80  Documents  (Policy/Directives/Statutes,  etc) 

-  122  IT  systems  and  features 

-  84  Strategic  elements  (goals  and  objectives) 

-  Over  600  functional  and  business  requirements 

-  Over  7,000  performance  factors  and  influencers 

Nearly  22,000  performance  relationships 

39 


vector/CSP 


Model  Configuration  Management 


Aviation 
Logistics  WPM 
Configuration 
Management 


@ 

Update 

The  WPM  database  is 

updated. 

© 

Change 

The  WPM  administrator 
makes  the  CCB-approved 
change  to  the  WPM. 


O 


CCB 


WPM  change  reviewed  and 
approved  via  CCB  and 
CG-22. 


[User  Review 


Using  the  WPM  web  page, 
users  review  jobs,  orgs, 

1  processes,  etc. 


Feedback 


User  notes  change  to 
process,  org  unit,  job  role, 

IT  system,  etc.  via  feedback 
form  located  on  each  page 
of  LPMS. 


Review 


LTPIO  evaluates  feasibility 
of  proposed  change.  If 
feasible  and  substantive, 
forward  to  CG-22  process. 


This  is  a  high-level  overview  of  the 
proposed  Aviation  Logistics  WPM 
configuration  management  process 
which  can  be  used  both  to  validate 
and  enhance  WPM  content  and  also 
to  manage  its  configuration  overtime. 
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People  are  Key 


•  Changing  the  behavior  of 
your  people  is  the  most 
difficult  task  of  all... 


•  Culture  and  politics  are  the  most  difficult 
roadblocks  to  logistics  transformation 


•  You've  got  to  convince  people  at  all  levels 
to  change  the  way  they  operate. 
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Promoting  cultural  and  political  change 


•  Each  outcome  identifies  human  factors 
influencers  (technological,  environmental, 
social,  and  process) 

•  Relationships  in  model  indicate  cultural  and 
political  power  bases 

•  Model  enables  stakeholders  to  "see"  vision 

•  The  Transformation  Dashboard  is  key 

to  producing  measurable  changes  in 
organization,  infrastructure,  processes, 
systems,  and  behavior. 

•  Manages  and  measures  progress  in 

making  the  changes  required  to 
achieve  “to-be"  outcomes 
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Transition  Success  Criteria 
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To-Be 

System 

Outcome 


Each  level  includes  a 
summary  scorecard  that 
shows  transition  progress. 


Performance 

Gap 


Performance 
I  ntervention 


External  audit  conducted  -  Final  Operating  Capability  (FOC)  achieved 
Internal  audit  conducted  -  Initial  Operating  Capability  (IOC)  achieved 
Process  requirements  identified  and  plan  of  action  in  place 
No  Progress 


43 


vector/CSP 


Example  Success  Criteria 


•  Outcome:  Authorized  Chemical  List 


-  I  intervention:  Chemical  Locker  Storage 
Established 

-  Criteria: 

•  Red:  No  Progress 

•  Orange:  Location  for  chemical  storage  locker 
identified. 

•  Yellow:  Authorized  Chemical  List  established  and 
utilization  instruction  developed  and  signed  by  Sector 
Engineering  Officer. 

•  Green:  Personnel  trained  in  proper  storage 
procedures  and  usage  of  the  Chemical  locker  I  AW 
Hazmat  plan. 

•  Owner,  schedule  criteria,  compliance  inspection 
process,  etc.  defined  for  intervention  in  model 
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Sample  USCG  Scorecard 
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Summary 


•  Performance  Model  Framework  identifies  all 
elements  of  logistics  system  performance 

•  All  relationships  between  systems  elements 
are  defined 

•  Performance  outcomes  are  central  to  model 

•  Each  outcome  has  transition  plan  that 
identifies  gaps  and  interventions 

•  Each  intervention  has  measurable  success 
criteria 

•  Success  criteria  form  the  basis  for  the 
Transition  Dashboard 

•  T ransition  Dashboard  drives  systems, 
organizational,  technological,  and 
behavioral  change 
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Demos  and  Questions 


•  USCG  Logistics  Performance 
Management  System  ( LPMS) 

•  Logistics  T ransition  Dashboard 

•  Transformation  support  materials 
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Introduction 


The  Army’s  vision  for  SMART  is  a 
process  In  which  we  capitalize  on 
Modeling  and  Simulation  (M&S) 
technology  to  address  the  issue  of 
system  development  and  life-cycle 
costs  through  the  combined  efforts 
of  the  requirements,  training  and 
acquisition  comm  unities. 


US  DoD  is  world’s  largest  single  consumer 
Simulation  Based  Acquisition 
Logistics  M&S  lags  engineering 

Increasingly  important  -  selection  discriminator 

v  r  ®  /  / 

n«i 


DoDD  5000.1  DoDI  5000.2 


Defense  Acquisition  Guidebook 

Definition  of  Simulation  Based  Acquisition  (SBA)/(SMART) 
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Importance  of  M&S 


Lessons  Learned  Through 
Acquired  Experience 

Apache 

(Legacy  system  upgrades  using  M&S) 

Crusader 

(Relied  heavily  on  M&S  to  support  systems  engineering) 

Comanche 

(Down  select  based  on  M&S) 

Grizzly 

(Revitalized  the  program  with  M&S) 

Aerial  Common  Sensor 

(Using  virtual  prototypes  for  source  selection) 

Future  Scout  and  Cavalry  System 

(Collected  Data) 


Army  Model  &  Simulation  Office 


MITRE  Technical  Exchange  Meeting,  October  9,  2001,  AMSO,  Policy  &  Technology  Division 
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Characterized  Levels  of  DoD  M&S 


2.  “Virtual”  (Man  vs  Simulation) 


3.  “Constructive” 


1.  “Live”  Man  vs  Man 


•  Training 

•  Development 

•  Test  &  Evaluation 
Operations 
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Static  (Steady  State)  &  Dynamic 
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No  dynamic  temporal  effects  .  Dynamic  temporal  effects 
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Raytheon 

Deterministic  &  Stochastic 


Totally  causal  events  &  decisions 
No  random  events 
No  “degrees  of  freedom” 


Indexed  collection  of  variables 
Random  number  generators 
Multiple  degrees  of  freedom 
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Raytheon 

Continuous  &  Discrete  Event 
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0,4as 


O.GlSS 


— ^Coordinator}< — i 

' 

Simulator 


Simulator 


) 

r 

Model 

Pi 

0  -  0US 


i.Ous 


Model 

wvvri 


Analog  computing  *  Time  referenced  events 

Differential  Equation  (partial/ordinary)  ’  Models  independent  of  simulators 
Continuous  between  f(x)  limits  *  Models  respond  to  events 

Approximated  by  digital  computers 


•  DEVS:  Formalization  =  less  errors 


Bernard  P.  Zeigler,  Herbert  Praehofer,  Tag  Gon  Kim, 

“Theory  of  Modeling  and  Simulation”,  2nd  Edition,  Academic  Press,  2000 
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Unitary  vs  Distributed  Simulation 


•  High  Level  Architecture  (HLA). 

•  Test  and  Training  Enabling  Architecture  (TENA) 

•  Aggregate  Level  Simulation  Protocol  (ALSP) 

•  Distributed  Interactive  Simulation  (DIS) 
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Military  Logistics 


•  Military  logistics  is  the  science  of  planning  and  carrying 
out  the  movement  and  maintenance  of  armed  forces.  In 
its  most  comprehensive  sense,  those  aspects  of  military 
operations  that  deal  with: 

-  a.  design  and  development,  acquisition,  storage,  movement, 
distribution,  maintenance,  evacuation,  and  disposition  of 
materiel; 

-  b.  movement,  evacuation,  and  hospitalization  of  personnel; 
acquisition  or  construction,  maintenance,  operation  and 
disposition  of  facilities;  and  acquisition  or  furnishing  of  services. 


Defense  Technical  Information  Center 
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Raytheon 

DoD  Acquisition  Framework 


User  Needs  & 
Technology  Opportunities 


•  Process  entry  at  Milestones  A,  B,  or  C 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step 


.  Concept 

Technology 

System  Development 

Production  & 

Operations  & 

:  8  Refinement 

Development 

&  Demonstration 

Deployment 

Support 

Concept 
!  ^/Decision 

A  Design 
<  >Readiness 

V  Review 

AFRP 

C  >  Decision 

V  Review 

if 


"  v 

b 


Graphics  Sourced  from  the  Defense  Acquisition  University 
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New  Cargo  Jet  Specification 
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Military  Logistics 

Defense  Acquisition  Guidebook  (Chap.  5.0) 


Total  Life  Cycle  Systems  Management 


1.  Supply  Chain 

2.  Manufacturing 

3.  Transportation 

4.  Storage 

5.  Deployment  i 


L 


Pre-System  Acquisition 

Plan  for  support 


Access  Support 
Considerations 


Plan  &  Refine 
Support 
Considerations 


Performance  Based 
Logistics 


(PBL) 


IOC 


FOC 


Production  & 
Deployment 


Operations  & 
Support 


Oi 


FRP 

Decision 

Review 


Sustainment 


Support  the  design 


Product  Support 
Design  Develop  &  Test 


TLCSM [:  The  PM  is  mandated  for  Life  cycle  Logistics(LCL),  emph 
Engineering  and  implementing  products  support  through  Perforrr 


Total  Life  Cycle 
System  Model 


Raytheon 


Integration/Manufacturing 

Civilian 


Deployed  Storage 

°  $ 

Military 
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Life  Cycle  System  Model 


Supply  Chain 


Supplier 


Component 

Integrator 


Supplier 


System 

Integrator 


Supplier 

■ 


Component 

Integrator 


Supplier 


Disposal 


Consumption 


■sSSSs*  , 

J^.hw 

L-jii.-imr,- 

■tarlwv 

■UlJIlB 

Raytheon 
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Supply  Chain 


Supplier 


Component 

1  Integrator 

System 

Integrator 

w 

,  Component 

Integrator 

— 


Customer 


W 


Manufacturing  Process 

Low  fidelity  models  -  parameterize  assumptions 
Obsolescence  management 
Socio-political  &  economic  effects 
KPPs: 

-  Consumption  driven  -  model  demand  drivers 

-  Cost,  capacity,  lead-time,  etc 


ML 
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Manufacturing  (Integration) 


Batch  and  Queue  Production 

ri 

1  | 

Product  Focused,  Single 

Piece  Flow,  Pull  Production  System 

V 


Customer 


■4  units  delivered 
for  production 


Willing 

Machine 


Assembly 

Machine 


Debarring 

Machine 


Chemical 

Treatment 

Machine 


Painting 

Machine 


Boring 

Machine 


Customer 


Parts 

Supplier 


Shipping 

Receiving 


Chemical 

Treatment 
Pept 


cal- 
em  in 

J— *  4 


Warehouse 

wrh- 

!'*r 


Milling 

Dept. 


Boring 

Dept, 


Painting 

Dept. 


Assembly 

Dept, 


Deburring 

Dept. 


1  w  \ 

Optimize  processes  &  material  flow 
Cost  (risk)  reduction 

-  Dynamic  -  Discrete  Event  Simulation 

-  Supply  Chain  model  integration 

KPPs 


k 


Or. 


-  Cost,  Process,  Flow 

-  Resource  Allocation,  etc. 
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Transportation 


Military  &  Civilian 
Touch  Labor 
Damage 
KPPs 

-  Reliability,  Availability 

-  Lead  Time  (customs,  etc.) 

-  Cost,  Choke  Points 

-  Sustainment,  Surge 


THE.  UNIVERSITY 
or  Arizona, 


Storage 


Location 

-  Depots  (Strategic) 

-  Theater  (tactical) 

KPPs 

-  Inventory  (operations,  spares) 

-  Cost,  availability,  reliability 

-  Assembly  ^  ^ 


\ri 


Raytheon 
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Deploy  &  Dispose 


*  Inventory  levels 

*  Asset  visibility 

*  Surplus  or  Destroy  ? 

*  Life  Extension 

-  Training 

-  Recycle 


Raytheon 
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M&S  Tool  Example 

Extend® 


|Replenishmer<Jri 


(Imagine  That  inc) 


Qn-bejrji  Storage 


Platforms 


i 

_ I 

HOTRFI  | 

|Fwl  From  Mi-ssi  onln  ^=^^  _  ^Thr^w 

I 

=4= 

k 

3  ^ — ^[kroarid  Signal  Out  | 


MissioriOrJ 


Tfjnipefl 


El 


insure 

Replenishment  NOT 

during  Production 


Infant  failures 
T  ransportation 
Storage  Environmental 
BIT  testing 
Training  use 


R*i  Vfr  he  on 
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M&S  Resources 


Military 

-  Defense  Modeling  &  Simulation  Office  (DMSO) 

-  Modeling  and  Simulation  Information  Analysis  Center  (MSIAC) 

-  Advanced  Air  Force  Modeling  &  Simulation  (AFAMS) 

-  Navy  Modeling  &  Simulation  Office  (NMSO) 

-  Army  Modeling  &  Simulation  Office  (AMSO) 

-  Army  Simulation,  Training  &  Instrumentation  (PEO  STRI) 

-  National  Security  Council  (NSC) 

Industry 

-  Simulation  Interoperability  Standards  Organization  (SISO) 

-  Association  of  Computing  Machinery  (ACM) 


Society  for  Modeling  and  Simulation  International  (SCS) 


/ 

Academia 

University  of  Arizona 
Georgia  Institute  of  Technology 
University  of  Pennsylvania 
INFORMS  College  on  Simulation 


ti  i  1/1 


Old  Dominion  University 
California  State  University 
University  of  Magdeburg 
University  of  Central  Florida 


ML 
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Conclusion 


M&S  is  increasingly  important  in  Military  Logistics 

-  Contractually  required  by  many  DoD  programs 
Beneficial  in  all  phases  of  life  cycle 

Highly  useful  in  logistic  modeling  &  management 

-  Highly  granular  (high  fidelity) 

-  Dynamic  inventories  &  environments 

-  Interoperable  with  engineering  &  operational  models 
Source  selection  discriminator 

-  Total  Lifetime  Cost  analysis  (TLC)  \ 

-  Logistic  support  capability 

/  \j 
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U.S.AIR  FORCE 


The  Deployment  Readiness  Service:  A  Case 
Study  of  the  Challenges  of  Implementing  a 
Service  Oriented  Architecture  in  a  Legacy 
System  Environment 

George  C.  Dalton  II,  USAF 
Air  Force  Institute  of  Technology 


Integrity  -  Service  -  Excellence 


Overview 


U.S.  AIR  FORCE 


■  What  Problem  is  being  solved  by  DRS? 

■  What  is  a  Service  Oriented  Architecture? 

■  What  are  the  Challenges? 

■  How  can  these  Challenges  be  overcome? 


The  views  expressed  in  this  presentation  are  those  of  the  author  and  do  not 
reflect  the  official  policy  or  position  of  the  United  States  Air  Force,  Department  of 
Defense,  or  the  U.S.  Government. 


Integrity  -  Service  -  Excellence 


What  Problem  is  being  solved  by  DRS? 


U.S.AIR  FORCE 


■  Unit  Deployment  Managers 

■  Must  pull  information  from  many  sources 

■  Personnel  Information 

■  Immunization  and  medical  records 

■  Training 

■  Etc.... 

■  Commanders  (at  all  levels) 

■  Need  a  snapshot  of  Unit  readiness 

■  Need  ability  to  Identify  problem  areas  and  trends 


Integrity  -  Service  -  Excellence 


w 


‘Vis  Is”  Readiness  Data  Challenge 

Example:  Training  Completion  Data  Origination  Process  (Generic) 


U.S.AIR  FORCE 


Student  Completes 
Course 


Authoritative  Source 
(Instructor) 
Creates  Completion 
Record 


Attendance  list  / 
Graduation  roster 


Specialized  System 


UDM/UTM 
Obtains  Copy  of 
record 


Fax/Copier 


Email 


Distribution 


msr 

Specialized  System 


In-Person 


Time  Flow 


UDM/UTM 
Manages 
Record  Copy 

Files  Copy  in  Mobilit 
Folder 


AND  (May) 

Manually  enter  some  data 
elements  from  record  into  one 
or  more  of  the  following 
systems: 


ACES-PR 


CAMS 


SFMIS 


LOGMOD 


G081 


ARMS 


Other  Functional  Systems 
(MRDSS,  LSA,  etc.) 


Local  Solutions 

(Excel,  Access,  Custom  Apps) 
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Integrity  -  Service  -  Excellence 


Deployment  Readiness  Service 

What  is  DRS? 

-  AF  personnel  deployment  readiness  solution 

-  Automates  a  manually  intensive  process 

-  Subscribes  to  data  from  multiple  sources 

-  Patlifinder  to  integrate  data  on  GCSS-AF 


Initial  Users: 

-  Commanders 

-  Installation  Deployment  Officers 

-  Unit  Deployment  Managers 

-  Unit  Training  Managers 

Why  DRS? 

-  Creates  standard  service  to  across  AF 

-  Integrates  diverse  readiness  indicators  (med,  trng,  etc.) 

-  Automates  review,  forecasting  (by  AEF),  and  scheduling 

-  Eliminates  dual  data  entry  (same  data;  multiple  systems) 

-  Eliminates  record  re-creation  (PCS/PCA) 
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Identify  Information  Needs 


U.S.AIR  FORCE 


Summary 

WSummaty 

CHARLES,  PIERRE  L. 
Requiiemeut  Status 


Rank:  MSG 
|/f  §1  B 


Role:  UDM 

Readiness  Overview 


Unit:  ,AFXX,WPAS 


STRAINING 

A 

Requirement  Red  Yellow 

Green 

RELIGIOUS_SENSmVlTY  5% 

FOREIGN. 

CLEARANCE  [Jj^J  2% 

86% 

SMALL.ARMS  gjgj  10% 

LOAC 

Eai  0% 

98% 

CHEM_WARF  ARE  0% 

98% 

CARGO_PREP  0% 

EOR 

II  8* 

100% 

FORCE 

°% 

SELF.AID 

0% 

_ jfl| 

AEF  Readiness  Overview 

|/}  |§]  H 

l 

A 

AEF  09,10  2 

3 

o  1 

o  1 

1  1 

I 

1 

AEF  03,04  9 

2  1 

0 

1 _ 1 

AEF  05,06  8 

0 

V 

_  Yellow  5 

12%  of  42 

Green  4 

L  ^ *•  A  I  0%  of  42 

2  Black  1 

2  %  of  42 


Flirjht  Readiness  Overview 


Unwaived  DAV  Code  Fixable/Expired 


YELLOW  GREEN 
Expiring  in  30  Days  Complete 


HELP  LOGOUT 


INDIVIDUAL  DEPLOYMENT  REQUIREMENTS 

DATE  PLACED  ON  DEPLOYMENT  STATUS 

DATE  ALL  ACTIONS  COMPLETED 

NAulE  (list.  First,  AWtfte  initial) 

STUFF, LEVEL  N. 

GRADE 

CPT 

AFSC 

UNIT/SHOP 

PRIMARY 

DEPLOYMENT  AVAILABILITY 

CODE  (DAV) 

REQUIRButENTS 


ID  CARD,  DD  FORM  2AFACT  (United  Staes  Unifonned  Services  ID 
CardXAcooumable  Form) 


DOG  TADS  (IDTaqs and  Osins) 


BvlERGENC Y  DATACARD.  DD  FORM 93  (tocord of  fiireqency  Qifa  GW) 


SHOT  RECORD,  PHS  73  1  , MwnaMwa)  of  Vaccinatm) 


LOCATOR  CARD,  AF  FORM  246  (Emplvymnt  tocaforawf  Free essinq  Oscblist) 


INSPECTION  RECORD 


Individual  Readiness  Summary  (IRS) 
Wireframe 


❖ 

U-S-  AIR  FORCE 


Logged  in  as:  Airferce,  Uiz  D. 

Rank:  TSgt  (E6) 

Login  Role:  UDM 

Login  Unit:  20  SVS,  20  FW,  Shaw  AFB 


“Requirement”  displays  solutions  c 

Requirement  STATUS 

PHA 

LAB 

SHOTS 

EQUIP 

DLC 


Equipment 


Requirement  S 
GAS_MASK 
GLASSES 
HEARING_AID 
PERSON  AL_CLOTHES 
PROF_EQUIP 
DOG_TAG 


Next  Requirement  Expires  20  May  2007 
Expires  11  Nov2o°7 


is  completed  to  satisfy  a  training  requirement, 

Requirement  S 

C ARGO_PREP 
CHEM_V/ARF  ARE 
EOR 
FORCE 

FOREIGN_CLEARANCE 

HAZ_CERT 

LOAC 

RELIGIOUS_SENSITIVITY 

SELF_AID 

SMALL_ARMS 


Requirement 

GENEVA_CARD 

GOV_DL 

LES 

LINE_BADGE 

PRP 

BAG_TAG 

HAND_RECEIP 

ID_CARD 

LOCATOR 


ig  has  been  completed. 

STATUS  Requirement  STATUS 

■  DEPENDENT 

■  POWER_ATT 


Click  on  box  to  initiate 
DAV  Code  query  for 
currently  displayed  person 
(waivers  can  be  applied 
from  query  results  screen) 


|  Click  here  to  create  AF  4005  with  displayed  data~|  |  Box  Color  Reflects  Category  Readiness"" |  |  Click  here  for  other  display  formats 


BLACK 


Jnwaived  DAV  CodelFixable/Expire 


YELLOW  GREEN 

| Expiring  in  15  Days 


Integrity  -  Service  -  Excellence 


Version  0.91  9  Apr  07  22 


Integrity  -  Service  -  Excellence 


Current  State 


f  ACES-PR  1 
l  ADS  J 

NIPRNET 


r  5 

{ 

>FMI5 

ADS 

>  l 

i 

Service-Oriented  Viewpoint 


U.S.AIR  FORCE 


Operational 

View 


Enterprise 

Services 

View 


Systems 

View 


In  a  Service-Oriented  Viewpoint  Operations  drive 
Services  and  Services  drive  Systems  -  effectively  decoupling 

Operations  from  Systems 


Integrity  -  Service  -  Excellence 


Future  State 


Readiness  Services 


W 


U.S.AIR  FORCE 


Deployment  Readiness  Service  (DRS) 

SV-4  Pathfinder  Authoritative  Data  Sources  Summary 


Organization 
Mgt  Data 


Q 


Task  Mgt 
ata 


(  Readiness 
\Folder  Data/ 


Course  Mgt 
Data  ___ 

Event  Mgt 
Data 


CReadiness^^x 
Eva[uationJ)afa 

/^Readiness^^x 

yV^BeportingJXata 


Organization  Data  Sources: 

•MilPDS  MPES 


Paper  Local  USAF  Joint 


Task  Data  Sources: 
•  DRS 


Readiness  Aggregation  Sources: 
•DRS  MilPDS  PIMR 


Course  Management  Data  Sources: 
•ACES-PR  SFMIS 

•ADLS  UDM  (DRS) 


Event  Data  Sources: 


Readiness  Evaluation  Data  Sources 
•  DRS 


Readiness  Reporting  Systems 
•  DRS 


Integrity  -  Service  -  Excellence 


Readiness  COI 

Preliminary  Authoritative  Data  Sources* 


ACES-PR  ■ 

■  Chemical  Warfare  Training 

SFMIS 

■  Arming  Group 

■  Small  Arms  Training 

MPES 

■  Position  Number 
PIMR 

■  Deployment  Limiting  Condition 

■  Equipment 

■  Immunizations 

■  Individual  Medical  Readiness 
Overall 

■  LAB 

■  Physical  Health  Assessment 
ADLS 

■  Explosive  Ordnance  Recognition 
(EOR)  Training 

■  Information  Assurance 

■  Language  and  Cultural  Training 

■  Law  of  Armed  Conflict  (LOAC) 
Self-Aid/Buddy  Care 

■  Trafficking  in  persons  Awareness 


MilPDS  ■ 

Air  Expeditionary  Force  Indicator 
Assigned  Flight  (office  symbol  if  flight 
unavailable) 

Assigned  Group 
Assigned  Installation 
Assigned  Squadron  or  Unit 
Assigned  Wing 

Control  Air  Force  Specialty  Code 
Duty  Air  Force  Specialty  Code 
Deployment  Availability  Codes 
Duty  Status  Code 
First  Name 
Gender 
Grade 

Grade  Description 
Last  Name 
Middle  Name 
Office  Symbol 

Primary  Air  Force  Specialty  Code 
Parent  Personnel  Accounting  Symbol 
Code 

Personnel  Accounting  Symbol  Code 
Social  Security  Number 
Special  Experience  Identifier  Code 
Special  Experience  Indicator  Type 
Suffix  (Name) 


UDM 

BAGGAGE  TAGS  (2  Sets) 

Date  Placed  on  Deployment  Status 
Dependent  Care  Certification,  AF  Fm  357 
Dog  Tags  (ID  Tags  and  Chains) 

Emergency  Data  Card  DD  Form  93 
Expeditionary  Combat  Skills  Training 
Force  Protection  Familiarization 
Foreign  Clearance  Guide  Briefing 
Gas  Mask  Spectacle  Inserts 
Geneva  Convention  Card,  DD  Form  1934 
Hand  Receipt,  AF  Form  1297 
Hazardous  Cargo  Certification  (If  Req'd) 

Hearing  Aids 
ID  Card 

Leave  &  Earnings  Statement,  DFAS  Fm  700 
Line  Badge,  USAF  Restricted  Area  Badge 
Locator  Card,  AF  Form  245  (Employment  Locator 
and  Processing  Checklist) 

Pallet  Build-Up/Cargo  Preparation  (If  Req'd) 
Personal  Clothing  Requirements 
Personnel  Reliability  Pgm  (PRP),  AF  Fm  286 
Power  of  Attorney 
Prescription  Glasses 
Professional  Equipment 
Religious  Sensitivity  Briefing 
Shot  Record,  PHS  73  1 
U.S.  Government  Driver's  License 
Will 


*  Initial  set  based  on  Spiral  0 
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Example  Use  Cases:  Log-in  For  All 
Use  Cases  and  Users 


U.S.AIR  FORCE 


■  User  logs-in  to  AF  Portal 

■  User  clicks  on  DRS 

■  DRS  comes  up  with  specific  role  for  current  user  based  on 
Portal  log-in 

■  Screen  shows 

■  User  summary:  %  of  unit  (or  wing)  ready,  #  overdue,  # 
expiring 

■  Print/export  to  Powerpoint  option 

■  User  name,  rank,  date,  unit  name,  base,  etc. 

■  Selectable  options 

■  Reports,  Query,  Create  (Requirements,  Notifications,  Thresholds), 
Mass  Actions  (Updates,  Printing),  My  Account,  etc. 


Integrity  -  Service  -  Excellence 


Example  Survey  based  off  of  Use  Cases 


Readiness  Reporting 

1 

Strongly 

Disagree 

2 

Disagree 

3 

Neutral 

4 

Agree 

5 

Strongly 

Agree 

Avg 

i.  DRS  provided  easy  to 
interpret  readiness  indicators 

0 

1 

2 

11 

9 

4.2 

ii.  DRS  accurately  produced  AF 
Forms  4005 

1 

3 

4 

10 

6 

3.7 

iii.  DRS  accurately  printed  AF 
Forms  4005 

0 

3 

6 

8 

6 

3.7 

iv.  DRS  enabled  me  to  conduct 
readiness  reporting 

0 

2 

7 

8 

4 

3.7 

v.  It  was  easy  to  notice  when 
there  were  problems  with  unit 
personnel  readiness 

0 

2 

i 

13 

7 

4.1 

vi.  DRS  enhanced  my  unit 
readiness  briefings 

0 

2 

7 

9 

3 

3.6 

Concept  Refinement 


U.S.  AIR  FORCE 


IOC 


FOC 


Concept 

Technology 

System  Development 

Production  a 

Operations  a 

Refinement 

Development 

ft  Demonstration 

Deployment 

Support 

\  Concept 
?  Decision 

A,  Oestjn 
f  J  Readiness 
\r  Review 

a  ™ 

f  j  Decision 
Review 

INPUTS 


*  ICD 

■  AOA  Plan 

*  Exit  Criteria 
■Alternative  Maintenance 

&  Logistics  Concepts 


OUTPUTS 


"Prelim  Sys  Spec 
"T&E  Strategy 
*S£P 

"Support  8l  Maintenance 
"Concepts  &  Technologies 
"Inputs  to: 

-draft  CDO 

-AoA 

-TDS 

-Cost/Manpower  Est 


Interpret  User  Needs, 
Analyze  Operational 
Capabilities  & 
Environmental  Constraints 


C' 


Analyzed  Assess 
Concepts  Versos 
Defined  User  Needs  & 
Environmental  Constraints 


i  TYades^ 


^Trades  j 


Develop  Concept 
Performance  (S  Constraints) 
Definition  ^Verification 
Objectives 


1  Trades^ 


Decompose  Concept 
Performance  into 
Functional  Definition  & 
Verification  Objectives 


Assess/Analyze 
Concept  &  Verify 
System  Concept’s 
Performance 


^Trades  j 


Assess/ Analyze 
System  Concept 
Versus  Functional 
Capabilities 


^  Trades^ 


^Trades  j 


Decompose  Concept 
Functional  Definition  into 
Component  Concepts  & 
Assessment  Objectives 


Ass  ess/ Analyze 
Enabling/Critical 
Components  Versus 
Capabilities 


Challenges 

•  Short  Schedule 

•  <  5Months 

•  Limited  Funding 

•  NTE  $1M 

•  No  additional  Funding  for  ADS 

•  Prescribe  Performance 

•  No  Data  Warehouse 

•  No  Pub/Sub 

•  Must  use  Request/Reply 

•  Must  Follow  COI  Vocabulary 

•  Not  yet  Developed 

•  Limited  Experience  Building  Services 


Develop  Component  Concepts, 
i,e,  Enabling/Critical 
Technologies,  Constraints 
&  Cost/Risk  Drivers 


*  Hard  to  map  services  in  DoDAF 


integrity  -  Service  -  Excellence 


Example  Original  Product  From 
Readiness  Community  of  Interest 

.S.  AIR  FORCE  * 


UKb  wuKKbnuP[J4j  -  UKS  strawman  kk  /  <Main  suoject  Area> 


Integrity  -  Service  -  Excellence 


Current  Process 


U.S.AIR  FORCE 


PREPARED 


19-SEP-2005  1520 


Personnel  Deployment  Checklist 


PCN  SA200U504 


INDIVIDUAL  DEPLOYMENT  REQUIREMENTS 

STATUS  DATE  ALL  ACTIONS  COMPLETED 

NAME  (Last,  First  Middle  Initial)  GRADE 

MARQUEZ,  RONALD  ANT  1LT 

DAFSC 

DUTY  SECTION 

PRIMARY 

021M3  MXMW 

DEPLOYMENT  AVAILABLITY 

00 

REQUIREMENTS 

COMPLETED 

INDIVIDUAL • S 

INITIALS 

INSPECTION  RECORD 

CHECKLIST  ITEMS 

ID  CARD  (DD  FORM  2AF) 

,c?<;^pnS 

Sf-m 

✓ 

LINE  BADGE  (AF  FORM  1199) 

fi  <ep  f'A 

?AR 

DOG  TAGS  (2  ID  TAGS  AND  1  LRG,  1  SM  CHAINS) 

KiJxi POS 

VAR 

/, 

EMERGENCY  DATA  CARD  (DD  FORM  93) 

Pi. 

?/?<« 

y. 

IMMUNIZATION  KECOKI}  ( DHHS  FORM  PHS  731! 

h  4-x  .  p  05. 

m*i 

y. 

CURRENT  LEAVE  AND  EARNINGS  STATEMENT  (DFAS  FORM  702) 

M  sfe-’p  05' 

sArr) 

l/ , 

US  GOVERNMENT  DRIVER'S  LICENSE  (AF  FORM  2293) 

fcI  SsJp 

iS 

WILL 

p  <-'~b  as 

RfiiP 

7, 

POWER  OF  ATTORNEY 

pSeP  cZ 

Wl  y 

1/ 

DEPENDENT  CARE  CERTIFICATION  (AF  FORM  357) 

MIR 

AJ//Z 

wt 

LOCATOR  CARD  (AF  FORM  24  5) 

y 

PRESCRIPTION  GLASSES  (2  PAIR) 

y 

GAS  MASK  SPECTACLE  INSERTS 

M  <n?-p  0*3 

y 

HEARING  AID  AND  2  SETS  OF  BATTERIES 

M/R 

U/fL 

j/ir 

PRESCRIPTION  MEDICATION  (180-DAY  SUPPLY) 

H  <f’~ 

V 

PERSONAL  CLOTHING/HYGIENE  ITEMS 

m-A 

y 

BAGGAGE  TAGS 

RAn/\ 

y 

ON-THE-JOB  TRAINING  RECORD  (AF  FORM  623) 

M/f 

A 

RM. 

PROFESSIONAL  EQUIPMENT 

B&n 

vj 

USAF  FIREARMS  QUALIFICATION 

in  te-jp  ftp 

RAM 

y 

GAS  MASK  FIT  CHECK 

n  aJP  ck 

PAdA 

y 

FAMILY  READINESS  BRIEFING 

Ki  < 

>.)Pc6 

tnn i 

y, 

POST/PRE  HEALTH  ASSESSMENT  FORMS 

A 

Ram 

i/ 

TRAINING  REQUIREMENTS 

, 

LAW  OF  ARMED  CONFLICT  (LOAC) 

77 

Far 

y. 

SELF-AID  BUDDY  CARE  (SABC) 

2  A  R 

/ 

FORCE  PROTECTION  FAMILIARIZATION 

my] 

y 

SMALL  ARMS  TRAINING 

l/y 

NUCLEAR-BIOLOGICAL- CHEMICAL  DEFENSE  (NBCDT)  AND 

EXPLOSIVE  ORDINANCE  RECOGNITION  (EOR)  TRAINING 

7 

RAM 

IMMUNIZATIONS 

. 

ORAL  POLIO  (ONE  DOSE  FOR  LIFE) 

27 -AUG-20 02 

RAM 

y 

MMR  (MUMPS /MEASLES/RD  BELLA) 

27-AUG-2002 

jZArt 

y 

TD  (TEMatOS/PIPHTHERIA) 

27-AUG-2002 

y. 

IPPD  <TB  SKIN  TEST  POST  DEPLOYMENT) 

2S-AUG-2002 

KaM 

y 

INFLUENZA  (ANNUALLY) 

07-FEB-2005 

iZAM 

y 

KBHOMUKE  (MENINGITIS)  (HIGH-RISK) 

27 -AUG-2002 

tAR 

y , 

HEPATITIS  A  (1ST  SERIES) 

27 -AUG-2002 

EfV* 

y 

HEPATITIS  A  (2ND  SERIES)  (DUE  6-12  MONTHS  LATER) 

27 -FEB-2003 

iZA'A 

y. 

YELLOW  FEVER  (TO  HIGH-RISK  AREA) 

2 5 -NOV-2 002 

s. 

TYPHOID  (TYPHIM  VI)  (2YR)  (TO  HIGH-RISK  AREA) 

2S-NOV-2002 

tm 

R 

ANTHRAX  #1  (INITIAL)  (HIGH-RISH) 

06-NQV-2Q03 

KM) 

y ' 

-TQA- 


-Page4of4- 


PERSONAL  DATA  -  BRXSSCY  ACT  GY  1974 
1SDWI0DM,  TRAINING  REQUIREMENT  NOTICE  AS  OF  17  OCT  OS 


NAME  EMP  #  GRADE  W/C  ORS  TOY  RETURN  BITS' 

MARQUEZ  RONALD  A  0QS63  014/  OUT  BOMB  / 


COTISS 

CODE  mts 

002986  CERTF15  IMPOUND  AUTHORITY 
006066  PADMINITI&L  EVAL 
081096  PADMMISSION  ORIENTATION 
006170  PADMH/C  CWTRQLLED  AREA  TNG 
010033  PMILORM  AWARENESS 
013024  PSECOPSEC 
013045  PSEGSECORin  TNG  PHASE  1 
©13046  K-ECSECORITY  TNG  PHASE  2 
018024  QANCLAS  OF  ARMED  CONFLICT 
018036  QANCPROT  FROM  TERROR  IVL  t 
018049  QANCSELF  AID  S  BUDDY  CARE 
018054  QARCSUICIDE  AlARE/VIOIiENCE 
018055  QANCSUPEHVTSOR.  SAFETY  TNG 
018071  QANCHOMOSEXQAL  POLICY  TNG 
iiilll  QDEPF5TR  CBRNE  DEFENSE  ETA 
019114  QDEPFSTR  CBRNE  TQT  STD  MR 
019242  QOEPSMMJ.  ARMS  9 MM  CAT  C 
024013  QMEDCPR  2  YEAR  RECERT 
027 04 9  QMONMONS  EXPLOSIVE  SAFETY 
A9F  05290  08:34:53  PROCESSED. 


DURATION 
7tB  iaw 
001  t 

001  s 

004  Y 
001  A 
002'  f 
001  A 
802  Y 
002  A 
001  H 
003  A 
ill  t 
001  B 
Q0.3  1 
001  J 
006  H 

002  a 

80S  t 
001  i 
002  A 


STATUS 

QOAL 

COMPL 

OOMSL 

COWL 

CDMPL 

COMPL 

COMPt 

COMPL 

QUAD 

QUSi 

QUAD 

QUAD 

QUAD 

QUAL 

*AKACT 

QUAD 

QUftL 

QUAL 

*A»ACT 


STATUS 

DATE 

12  g.0V  Q4 
17  3M  03 
:  FE8  03 

i  JAN  03 


If  OCT  tffi 

3 

30  JAN  86 
30  APR  06 
08  FEB  Q6 
25  NOV  G6 
08  AUG  06 


07  JAN  06 
17  JON  06 
01  OCT  07 
25  NOV  06 
16  NOV  05 
I  DATE:  MfllO'5 


CAMS  Report 


INDIVIDUAL  DEPLOYMENT  RETIREMENTS 


REQUIREMENTS 


DATE  INITIALLY 
ACCOMPLISHED/ 
COMPLETED 


DOCUMENTATION 


DOG  TAGS  (ID  Tags  and  Chains) 


EMERGENCY  DATA  CARD,  DD  FORM  93  (Record  of  Emergency  Data  Ca 


SHOT  RECORD,  PHS  731  (Inte 


j  si  Certificate  of  Vaccination ) 


LOCATOR  CARD,  AF  FORM  245  (Employment  Locator  and  Processing  Checklist j 


BAGGAGE  TASS  (2  Sets) 


GENEVA  CONVENTIONS  CARD,  DD  FORM  1 934  (Geneva  Conventions  identity 
Card)  FOR  MEDICAL  AND  RELIGIOUS  PERSONNEL  WHO  SERVE  IN  OR 
ACCOMPANY  THE  ARMED  FORCES 


PRP,  AF  FORM  286  (Perso 


DEPENDENT  CARE  CERTIFICATION,  AF  FORM  357 


PRESCRIPTION  GLASSES  (2 Pairs)  (1  pairforARC) 


GAS  MASK  SPECTACLE  INSERTS 


HEARING  AIDS 


PROFESSIONAL  EQUIPMENT 


LINE  BADGE,  USAF  RESTRICTED  AREA  BADGE  (Ascot 

OPTIONAL 


3.  GOVERNMENT  DRIVER'S  LICENSE  ( Special  Purpose  Vehicles) 


CLOTHING  REQUIREMENTS 


LES,  DFAS  FORM  700  (Leave  and  Earning  Statement) 


DATE  ALL  ACTIONS  COMPLETED 


INSPECTION  RECORD 


SELF-AID/BUDDY  CARE 


PALLET  BUILD-UP/CARGO  PREPARATION  (If  Requin 


HAZARDOUS  CARGO  CERTIFICATION  (If  Required) 


RELIGIOUS  SENSITIVITY  BRIEFING 


3N  CLEARANCE  GUIDE  BRIEFING 


EXPLOXIVE  ORDNANCE  RECOGNITION  (EOR)  TRAINING 

LAW  OF  ARMED  CONFLICT  (LOAC)  BRIEFING 


FORCE  PROTECTION  FAMILIARIZATION 


INDIVIDUAL'S  SIGNATURE 


UDM/BUPERVOR'S  SIGNATURE 


AF  FORM  4005,19970601  (EF-V2) 


REPLACES  ACC  FORM  160,  MAR  96,  WHICH  IS  OBSOLETE. 


AF  FORM  4005,  JUN  97  (EF-V1) 


REPLACES  ACC  FORM  160,  MAR  96,  WHICH  IS  OBSOLETE. 


LOGMOD  4005 


Blank  4005 
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Integrity  -  Service  -  Excellence 


Form  4005  Source  Matrix 


U.S.AIR  FORCE 


Data  Field 

Data  Provider 

Method  Of  Entry 

Top  Portion 

• 

Date  Placed  on  Deployment  Status 

UDM** 

Manual  Entry  in  DRS 

• 

Date  All  Actions  Completed 

UDM** 

Manual  Entry  in  DRS 

• 

Name 

MilPDS 

Automated  Feed 

• 

Grade 

MilPDS 

Automated  Feed 

Documentation 

• 

ID  Card,  DD  Form  2AFACT 

UDM** 

Manual  Entry  in  DRS 

• 

Dog  Tags 

UDM** 

Manual  Entry  in  DRS 

• 

Emergency  Data  Card,  DD  Form  93 

UDM** 

Manual  Entry  in  DRS 

• 

Shot  Record,  PHS  731 

UDM** 

Manual  Entry  in  DRS 

• 

Locator  Card,  AF  Form  245 

UDM** 

Manual  Entry  in  DRS 

• 

Hand  Receipt,  AF  Form  1297 

UDM** 

Manual  Entry  in  DRS 

• 

Baggage  Tags 

UDM 

Manual  Entry  in  DRS 

Other  (if  Required) 

• 

Geneva  Conventions  Card,  DD  Form  1934 

UDM 

Manual  Entry  in  DRS 

• 

PRP,  AF  Form  286 

UDM 

Manual  Entry  in  DRS 

• 

Dependent  Care  Certification,  AF  Form  357 

UDM 

Manual  Entry  in  DRS 

• 

U.S.  Government  Driver’s  License 

UDM 

Manual  Entry  in  DRS 

• 

Prescription  Glasses 

UDM 

Manual  Entry  in  DRS 

• 

Gas  Mask  Spectacle  Inserts 

UDM 

Manual  Entry  in  DRS 

• 

Hearing  Aids 

UDM 

Manual  Entry  in  DRS 

• 

Personal  Clothing  Requirements 

UDM 

Manual  Entry  in  DRS 

• 

Professional  Equipment 

UDM 

Manual  Entry  in  DRS 

• 

Line  Badge,  USAF  Restricted  Area  Badge 

UDM 

Manual  Entry  in  DRS 

Optional 

• 

LES,  DFAS  Form  700 

UDM 

Manual  Entry  in  DRS 

• 

Will 

UDM 

Manual  Entry  in  DRS 

• 

Power  of  Attorney 

UDM 

Manual  Entry  in  DRS 

• 

Blank  Fields 

UDM 

Manual  Entry  in  DRS 

Training 

• 

Small  Arms 

SFMIS 

Automated  Feed 

• 

Self- Aid/Buddy  Care 

ADLS 

Automated  Feed 

• 

Chemical  Warfare 

ACES 

Automated  Feed 

• 

Pallet  Build-Up/Cargo  Preparation  (If  Required) 

UDM 

Manual  Entry  in  DRS 

• 

Hazardous  Cargo  Certification  (If  Required) 

UDM 

Manual  Entry  in  DRS 

• 

Religious  Sensitivity  Briefing 

UDM 

Manual  Entry  in  DRS 

• 

Foreign  Clearance  Guide  Briefing 

UDM 

Manual  Entry  in  DRS 

• 

Explosive  Ordnance  Recognition  (EOR)  Training 

ADLS 

Manual  Entry  in  DRS 

• 

Law  of  Armed  Conflict  (LOAC)  Briefing 

ADLS 

Automated  Feed 

• 

Force  Protection  Familiarization 

ADLS 

Automated  Feed 

Integrity  -  Service  -  Excellence 


Cross-check  Model  for  Completeness  &  Consistency 


U.S.AIR  FORCE 


Integrity  -  Service  -  Excellence 


Deployment  Readiness  Service 

Readiness  Items  Grouped  by  Type 
Example 


Common  Evaluation  Areas 
Common  Criteria,  AF  Form  4005 
Common  Criteria,  Non-AF  Form  4005 


Readiness 


Optional  Item 
Duty  Specific  Criteria 
Conditional  Criteria 


AEF  Task  Specific  Criteria 
Functional  Task  Criteria 
Unit  Task  Specific  Criteria 


r 

( 

\ 

( 

\ 

Administrative 

Equipment 

Legal 

Medical 

Training 

Readiness 

Readiness 

Readiness 

Readiness 

Readiness 

V _ 

J 

_ 

J 

v _ 

) 

v _ 

J 

v _ 

J 

X 


I 


AF 

Manual 

100 


Duty 

Status 


X 


LES, 
DFAS 
Fm  700 


I 


U.S.  Gov’t 
Drivers 
License 


X 


AEF 
Assignment 
Letter 


I 


ID 

Card 


Emergency 
Data  Card 
DD  Fm  93 


T 


1 


Time 

DAV 

Code 


Admin 

DAV 

Code 


I 


1 


MyPay 

Access 


Security 

Clearance 


I 


Baggage 

Tags 


1 


PRP 
AF  Fm 
286 


_ i _ 

_ i _  _ i _ 

Geneva 

Convention 

Card 

r  \ 

Courier 
Letter 

AF  Fm 
4005 

Hand 
Receipt 
AF  Fm 
1297 


I 


Dog 

Tags 


Prescription 

Glasses 


Hearing 

Aids 


I 


I 


1 


Canteen 


Gas  Mask 
Spectacle 
Inserts 


Personal 

Mobility 

Bag 


T 


1 


Flak 

Vest 


Bug 
Spray  J 


Helmet 


Weapon 


I 


Ear 

Plugs 


1 


Professional 

Equipment 


.  -t  . 

Dependent 
Care  Plans 
AF  Fm  357 


Shot 

Record 


PHA 

Review 

Dental 


Lab 


522 

Profile 


Immunization 

Status 


f - \ 

Medical 

Equipment 

Dental 

Clearance 


Medical 

Clearance 


X 


I 


Yellow 
Fever  Shot 


90  Day 
Prescription 


NBC 

Defense 

Training 


Law  of 
Armed 
Conflict 


Self-Aid 

Buddy 

Care 


Red  X 
Training 


Information 

Assurance 


Will 


Life 

Insurance 
Election 
SGLV  8286J 


''Pre-Deployment  ^ 

Health 

Small 

,  Assessment  > 

Arms 

^  J 

| 

— v — — ^ 

/ 

i 

— x  r- L 

M16 


M9 


Foreign 

Clearance 

Guide 

Briefing 


EOR 


Training 


Hazardous 

Cargo 


Force 

Protection 

Training 


Weapons 

Courier 


Pallet 

Build 


Religious 

Sensitivity 

Briefing 


Note:  This  is  by  no  means  a  complete  set  but  provides  insight  to  the  types  of  factors  involved 
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Other  Checks 


w 


U.S.AIR  FORCE 


Organization 


Person 


Reporting 


Readiness 

Requirements 


Medical 

Equipment 

Training 

Admin 

Legal 

Source: 


I  Sh 


Shot  Rec 
Dental 
HRR 


Glasses 

Lab 

Overall  Stat 


Readiness 

Tasks 

(e.g.  items  completed  or  inspected) 


Doq  Taqs 
Jjearing  Aids 
Pro  Equip 


Gas  Mask  Inserts 
Clothing  Req’ts 
Custom  Req’t 


~ 


Shot  Rec 

Glasses 

Imm.  Stat. 

Dental 

Lab 

>  Profile/422 

HRR 

Overall  Stat 

Actions 

BLACK  RED 


Unwaived  DAV  Code  Fixable/Expired 


Shot  Rec  Glasses 


Doq  Taqs 
Hearing  Aids 
Pro  Equip 


Time  DAV  Ha 
Admin  DAV  L 
ID  Card  B 
Emer.  Data  Gc 

PoA 

Dep  Care 


Hand  Recpt 
Locator 
iBaqiW^li 
Geneva  Cd 


PRP 

Gov’t  Lie. 
Line  Badge 
LES 

Will 

LDAV 


Small  Arms' 
Pallet 


Gas  Mask  Inserts 
Clothing  Req’ts 
Custom  Req’t 


NBC 


SABC  I 


Haz  Cargo 


EOR 


Time  DAV  1  Hand  Recpt  1 

— 

Admin  DAVl|  Locator 

Gov’t  Lie. 

ID  Card  1  Bag  Tags 

Line  Badge 

Emer.  Data  \  Geneva  Cd  1 

LES 

PoA  T 

Will 

Dep  Care  L 

.DAV 

YELLOW  GREEN 


Dental 


HRR 


Lab 


Overall  Stat 


Imm.  Stat. 


Profile/422 


Actions 


Doq  Taqs 
Hearing  Aids 
Pro  Equip 


Gas  Mask  Inserts 
Clothing  Req’ts 
Custom  Req’t 


Time  DAV  Hand  Recpt 
Admin  DAV  Locator 
ID  Card 

Emer.  Data  Geneva  Cd 

PoA 

Dep  Care 


ll_  PRP 
Gov’t  Lie. 
Line  Badge 
LES 

Will 

LDAV 


ADLS 


MilPDS  I  SFMIS 


DRS 


PIMR 


AFCITA 


Integrity  -  Service  -  Excellence 


§yibita§iki.=P’l 


w 


UDM  Selects  Airman  &  Requests  Indiv  Readiness  (4005  +  Medical  Data) 

U.S.AIR  FORCE  Use  Case  references  X-025 


USER(UDM) 


User  needs  to  review 
person's  AF  Fm  4005. 
Selects  person  in  unit 


DRS  Web  App 
DRS  Server  Apps 


Org  structure  expands  to 
review  sub  units  & 
people.  Person  selected 


- - 

Build  request  for  person  info, 
readiness  req'ts  and  items  to 
complete  4005 


User  views  welcome  screen 
with  their  name  and  role 


e.g.  calls  server 
app  (4005 
Service  / 
function) 

e.g.  creates  request  for  person  info 
(grade,  AFSC,  DAV  codes),  readiness 
requirements,  readiness  items 


User  views 
Airman's  4005 


Displays  aggregated  data 
in  requested  4005 
presentation  format 


Receives/ 
checks  requested 
aggregate  object 


AF  Portal 
Aggregation  Service 

(BPEL  driven) 


ADS  (Organizat 


ADS:  MilPDS 
Grade,  AFSC, 
DAV  Codes 


Execute  request;  splits 
into  individual  ADS 
requests 


Receive  list  of  req'd  items, 
Identify  ADS  for  each  req'd 
item  and  request  item  status 


Collect  responses,  match 
item  status  to  requirement, 
build  aggregate  object 


Name,  Grade,  Unit, 
DAV  status,  req't  list 
matched  with  status 


MDE  Used  to  discover  at  design//e$^<^a§i€rft^JiYie  Service  -  Excellence 


Example  Sequence  Diagram 


W 


U.S.AIR  FORCE 


UDM  Requests  Individual  Readiness  Sequence  Diagram 


DRS  Web  Add 


Aggregation  Engine  Service 


DRS  ADS 

SFMIS  ADS 

MILPDS  ADS 

ADLS  ADS 

ACES-PR  ADS 

PIMR  ADS 

<User  Initialization  Process> 
UDM  Selects  (clicks  on)  Individual 


Get  Ind  Readiness(SocialSecurltyNumber) 


(Pass  Aggregation  model ) 
Get  Ind  Readiness(Mode)  Param) 


Required 
Read mess  Data 
that  are  not 
otrwrvw  $9 


Parameters  to 
each  ADS  aa 


Get  DRS  Readiness  Requirements  (SSN) 

- - - - V. 


DRS  Readiness  Requirements 


Get  Weapons  Training  Readiness  Requirements  (SSN) 

- — — ! - H 

Weapons  Training  Readiness  Requirements 

4 - j - 


Readiness  Business  Object  or  Error 
- L> 


Get  Readiness  Items  (SSN) 


DRS  Readiness  Items  or  error 


Get  Weapons  Training  Readiness  Items  (SSN) 


Weapons  Training  Readiness  Items  or  error 


Get  Personnel  Readiness  Items  (SSN) 


Personnel  Readiness  Items  or  error 

Get  Ancilliary  Training  Readiness  Items  (SSN) 


Ancillary  Training  Readiness  Items  or  error 


Get  Chem  Warfare  Readiness  Items  (SSN) 


Chem  Warfare  Readiness  Items  or  error 


Get  Medical  Readiness  Items(SSN) 


Medical  Readiness  Items  or  error 


Working  DRAFT 


Integrity  -  Service  -  Excellence 


DRS  develop/deploy/use  Timeline 

UC  11:  Use  Mission  Services  (Web  App) 


Note:  Aggregation  service  does  not  use  MDE  to  determine  which 
Mission  Services  (Wrappers)  to  call  at  runtime 


[ 

r  * 

1 _  Ln 

;igr 

i 

D 

ev< 

elo 

P 

[ 

ar 

_ 

)lcr 

/ 

E 

xe< 

:ut 

e 

► 


ia^j.MncniWiiHaH 


MDE-SOA  Environment  Primer 


Integrity  -  S e PWi&e  -  Excellence 


DRS  Pathfinder  SV-1  (DRAFT) 


w 


U.S.AIR  FORCE 

o  =  Mission  Service 
o  =  Enterprise  Service 
CDS  =  Core  Data  Service 

VDE  =  Vocabulary  Defined 
Elements 


ADS 

DB 

ADS 

Transactional 

Processes 


Integrity  -  Service  -  Excellence 


Refine  the  Model 


U.S.AIR  FORCE 


Readiness  Services 
Data  Model  Construct 


Update  DoD  Architecture  Framework  (DoDAF)  artifacts 


U.S.  AIF!  FORCE 


All  data  relating  to  a  planned, 
required  or  notional  task; 
including  required  resources, 
time,  competencies,  etc. 


All  data  relating  to  or 
attributable  to  a  specific 
person.  It  includes  data 
related  to  assignment  of 
organization,  tasks,  events, 
and  completion  of 
readiness  requirements 
such  as  training. 


All  data  related  to  the 
hierarchy  of  formal, 
functional,  ad  hoc,  or 
notional  organizations; 
including  assigned 
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Framework  Product  Name 

Overview  and  Summary 

I  Information  ■ 


Integrated  Dictionary 


High-Level  Operational 

Concept  Graphic  _ 

Operational  Node  Connectivity 

Description 


Operational  Information 

Exchange  Matrix 


Organizational  Relationships 

Chart 

Operational  Activity  Model 


Operational  Rules  Model 


Operational  State  Transition 

Description  _ 

Operational  Event-T race 

Description  _ 

Logical  Data  Model 

Systems  Interface  Descnption 


Systems  Communications 

Description 


Deployment  Readiness  Service  (DRS) 
Operational  View  (OV-1) 

Readiness  Activity  Centers 
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General  Description 


Scope,  purpose,  intended  users,  environment  depicted 

analytical  findings 


Architecture  data  repository  with  definitions  of  all  terms  used  in 

all  products  '  _ 

High-level  graphical/textual  descnption  of  operational  concept 


Operational  nodes,  connectivity,  and  information  exchange 

needlines  between  nodes 


Information  exchanged  between  nodes  and  the  relevant 

attributes  of  that  exchange 


Organizational,  role,  or  other  relationships  among 

organizations 

Capabilities,  operational  acti  vities,  relationships  among 

activities,  inputs,  and  outputs;  overlays  can  show  cost, 
performing  nodes  or  other  pertinent  information 


_  >ther  pertinent  information 

One  of  three  products  used  to  descnbe  operational  activity- 

identifies  business  rules  that  constrain  operation 

One  of  three  products  used  to  descnbe  operational  activity— 

identifies  business  process  responses  to  events 


One  of  three  products  used  to  descnbe  operational  activity — 

traces  actions  in  a  scenano  or  sequence  of  events 

Documentation  of  the  system  data  requirements  and  structural 

business  process  rules  of  the  Operational  View 


Identification  of  systems  nodes,  systems,  and  system  items 

and  their  interconnections,  within  and  between  nodes 

Systems  nodes,  systems,  and  system  items,  and  their  related 

communications  lay -downs 


Relationships  among  systems  in  a  given  architecture;  can  be 

designed  to  show  relationships  of  interest,  e.g.,  system-type 
interfaces,  planned  vs.  existing  interfaces,  etc. 


Functions  performed  by  systems  and  the  system  data  flows 

amonij  system  functions 


Mapping  of  systems  back  to  capabilities  or  of  system  functions 

back  to  operational  activities 


Provides  details  of  system  data  elements  being  exchanged 

between  systems  and  the  attributes  of  that  exchange 

Performance  charactenstics  of  Systems  View  elements  for  the 

appropriate  time  frame(s) 

Manned  ir  ... 


incremental  steps  toward  migrating  a  suite  of  systems 

to  a  more  efficient  suite,  or  toward  evolving  a  current  system  to 
a  future  implementation 


Emerging  technologies  and  software/hardware  products  that 

are  expected  to  be  available  in  a  given  set  of  time  frames  and 
that  will  affect  future  development  of  the  architecture 


One  of  three  products  used  to  describe  system  functionality — 

identifies  constraints  that  are  imposed  on  systems  functionality 
due  to  some  aspect  of  systems  design  or  implementation 


One  of  three  products  used  to  describe  system  functionality — 

identifies  responses  of  a  system  to  events  _ __ _ 

One  of  three  products  used  to  describe  system  functionality— 

identifies  system- specific  refinements  of  critical  sequences  of 
events  described  in  the  Operational  View 

Physical  implementation  of  the  Logical  Data  Model  entities, 

e.g.,  message  formats,  file  structures,  physical  schema 
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Readiness  Services 


Identify  Information  Assets 
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{  Readiness 

Task  Data  ) 


Organization  Data  Sources: 
MilPDS  I  DCPDS  Local  D 


Paper  Local  USAF  Joint 


Task  Data  Sources  (automation  incomplete): 

•  AFIs  UDM  spreadsheets  TPFDD  libraries 

•  Emails  Web  pages  Phone  Local  policy 


Readiness  Data  Sources: 

•  LOGMOD  CAMS/IMDS 
ARMS 


ACES-PR 

mIlpds  ~ 

PIMR 


SFMIS 

DRS 


G081 
MRDSS 


DEERS 

AFCITA 


Paper 
Artifacts 

ADLSJUDM  Spreadsheets 
DCPDS  DIMHRS 
DM HRS  i 


Course  Management  Data  Sources: 

•Training  spreadsheets  Paper  rosters 
•  Local  Tools  Dozens  of  MAJCOM  systems 

Emerging  tools*:  DIMHRS,  ADLS,  SRRS,  DRS,  etc. 


Event  Data  Sources: 

•  Staff  meeting  charts 


Outlook  calendar  Local  Tools 


Readiness  Evaluation  Systems 

•  Mobility  folder  reviews  UDM  Spreadsheets 

•  AF  Form  4005  Modified  AF  Form  4005 


|  DRS 


Readiness  Reporting  Systems  (Group  or  higher)  DRS  | 
•  Manual  aggregation  and  manipulation  of  local  data 


No  “as  is  source”  for  enterprise  Coordination  &  Orchestration  data 
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w  Individual  Readiness  Summary  (IRS) 
*1*  Wireframe 


Map  Data  Flow 


Use  Case  #X-022 

U.S.  Al R  FORCE  (Use  Case  X-010  is  prerequisite) 


USER 


Activated  by  selecting  an  individual  from  an  actionable  link 
from  other  users  cases-including  but  not  limited  to  X-020, 
X-030,  X-032,  X-035,  X-036,  X-040) 


DRS  Web  App 


Calls  IRS 
service/function  with 
selected  user’s  ID _ 


e.g.  calls  server 
app  (IRS) 


DRS  Server  Apps 


Display  Fields: 

•  See  IRS  wireframe 


Screen 
Data  Object 


Actions: 

•  Sort  by  display  fields 

•  Export  to  AF  4005 

•  Launch  DAV  code  query  for 
user 


User  views  selected  Airman’s 
IRS  can  change  viewing 
format  or  generate^4005 


Displays  aggregated 
data  in  IRS  presentation 
format _ 


Build  request  for  person  info, 

e.g.  creates  request  for  person 

•  Req’t  drill  down  (future)  t/ 

Receives  / 

readiness  req’ts,  eval  criteria, 
and  items  necessarv  to  build  IRS 

irnu  ^yraue,  rtroc,  umv  cuues;, 
readiness  requirements, 
readiness  tasks  (completion  data) 

checks  requested 
aggregate  object 

Aggregation 

Service 


Use  session  data,  identify  and 
obtain  any  missing  pieces 
through  individual  ADS  requests 


ADS  (Definitions  &  Catalogs) 


(Mar 


DRS:  Get_DAV_Code_Definitions 
DRS:  Get_AEF_Definitions 
DRS:  Get_Requirement_Definitions 
DRS:  Get_Readiness_Status_Definitions 
DRS:  Get  ADS(Null) 


ADS  (Organization) 


|  MILPDS:  Get Detailed lndividual Data 

ADS  (Requirements) _ 

I  DRS:  Get  Assigned  Requirements  


Receive  list  of  req’d  items, 
definitions  &  task  sources, 
and  request  task  status 


Collect  responses,  match 
item  status  to  requirement, 
eval,  build  aggregate  object 


Name,  Grade,  Unit, 
DAV  status,  req’t  list 
matched  with  status 


Request  for  DAV 
codes,  titles,  ADSs, 
requirements,  tasks 


Request  for 
person  info 


SFMIS:  GetArmingGroup 


Request  for 

Applicable 

Requirements 


ADS  (Readiness  Evaluation) _ 

fPRS:Get  Readiness  Eval  Threshold!! 

I  DRS:  Get  DAV  Code  Waiver 


Request  for  evaluation 
thresholds 


|  DRS:  Store Readiness Evaluation 


Evaluation  results  stored 
as  last  eval  record 


ADS  (Readiness  Tasks) 


|  SFMIS:  Get  Completed  Training 

I  ACES-PR:  Get  Completed  Training 


iRe 


quest  for  Small  Arms  Trng 


qiest  for  CBRNE  Training 


Request  for  other  Ancillary  Training 


|  ADLS:  Get  Completed  Training 
PI  MR:  Get  Medical  Readiness  Indiyjdual^^rReqLt  pSt  for  Medical  Eval  Status 
DRS:  Get Readiness item  Request  for  other  required  items" 
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DoDAF  Service  View 
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Figure  5-24:  Notional  Example:  SV-4b:  Data  Flow  Diagram 
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Overcoming  Challenges 


U.S.  AIR  FORCE 


•  Realistic  Schedule 

•  Realistic  Funding 

•  Rethinking  Performance  Constraints 

•  "Regional  Cache" 

•  Up  Front  Engineering 

•  Created  Technical  Package 

•  Iterative  Vocabulary  Development 
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Summary 


U.S.  AIR  FORCE 


■  What  Problem  is  being  solved  by  DRS? 

■  What  is  a  Service  Oriented  Architecture? 

■  What  are  the  Challenges? 

■  How  can  these  Challenges  be  overcome? 
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Questions? 
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R-TOC  Genesis 


•  Initiated  in  1999  by  the  USD(AT&L)  to  address: 

-  O&S  cost  growth  at  expense  of  force  modernization  and 
readiness 


-  O&S  budget  constraints  limit  programs  to  near-term,  critical 
solutions  only 

-  R-TOC  program  seeks  to  seed  O&S  cost  avoidance 
solutions  that  have  broader  impact 


-  Thirty  Pilot  Programs 


Rising 
O&S  Cost 


Less  $  for 
Modernization 


Aging 

Equipment 


Fixed  Top  Line 


Declining  Future 
Readiness 


DEATH  SPIRAL 


USD(AT&L)  FY  2005  R-TOC  Goal 


USD(AT&L)  Goal:  .  .reduce  the  O&S  cost  of  fielded 
systems  (excluding  manpower  and  fuel)  by  20% 

(compared  to  current  FY  1998  levels)  by  the  year  2005.” 

“Overall,  each  Service’s  O&S  reduction  plans  will  be  based 
on  tradeoffs  among  these  three  areas  for  savings: 

1 .  Reduced  demand  from  weapon  systems  via  reliability 
and  maintainability  improvements 

2.  Reduced  supply  chain  response  times,  leading  to 
reduced  spares,  system  support  footprint,  and  depot 
needs 

3.  Competitive  sourcing  of  product  support,  leading  to 
streamlining  and  overhead  reductions” 


FY  2005  O&S  Savings 


•  FY  2005  cost  avoidances  exceeded  $2.1  B 

•  Projected  life  cycle  cost  avoidances  will  exceed 
$76B,  for  the  R-TOC  Pilot  Programs 

O&S  Costs  Can  Be  Reduced!! 


Life  Cycle  Savings  Provides  a  Focus 
on  Long  Term  Benefits 


New  Strategic  Direction 


•  With  the  successful  completion  of  the  Pilot  Programs 
FY  2005  goal,  a  new  direction  was  needed 

•  Strategic  Directions: 

-  New  goal  for  FY  2010 

-  Focus  on  life  cycle  O&S  cost  reductions 

-  Direct  funding  for  long-term  savings  projects 

-  Adoption  of  quantitative  method  for  evaluating 
near-term  vs.  long-term  projects 


USD(AT&L)  FY  2010  R-TOC  Goal 


•  USD(AT&L)  Goal:  “Maximize  cost  avoidance  on  total 
defense  systems  FY  2010  O&S  costs  from  an  FY 
2004  baseline,  by  offsetting  30%  of  predicted 
inflation.” 

-  Goal  extends  to  all  defense  systems  on  program-by- 
program  basis 

-  15  Special  Interest  Programs  (SIPs)  designated  lead 
programs  to  “show  the  way”  towards  achieving  the  goal 

-  SIPs  are  monitored  through  semi-annual  reports  and 
quarterly  R-TOC  Forums 

-  Services  will  include  this  goal  in  their  reviews 

•  Ultimately  expand  to  all  defense  systems 

•  $25M/year  R-TOC  PE  created 


Status  of  R-TOC  SIP  Program  Savings 
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UH-60M  Composite  Tailcone 


Program  Description 

Problem:  The  currently  proposed  metal 
tailcone  for  the  UH-60M’s,  MH-60S’s 
and  MH-60R’s  are  labor  intensive  to 
manufacture  and  require  thousands  of 
parts  and  fasteners. 

Solution:  Incorporate  a  composite 
tailcone  into  the  UH-60M,  MH-60S  and 
MH-60R  fleets. 


Benefits 


Investment/ROI 


•  Cost  savings  of  $60,000.00  per  new 
production  aircraft. 

•  Fewer  parts  and  fasteners 

•  No  corrosion  or  fatigue  maintenance 

•  Weight  Reduction  (50  pounds) 


Investment:  $2.35M 
Life  Cycle  ROI:  33:1 
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Ship’s  Material  Condition  Model 


Overview/Problem 

■  USN  does  not  have  a  consistent  objective 
method  to  determine  material  condition  and  its 
impact  on  mission  /  warfare  area 

■  USN  has  multiple  antiquated  software  tools  and 
systems  to  validate,  screen  and  broker  work 
candidates  depending  on  platform  type  and  coast 

■  USN  has  no  objective  method  to  determine  future 
material  condition  readiness  when  routine 
maintenance  is  not  performed 


Solution 


Investment/ROI 


■  Model  each  ship  using  a  hierarchical  structure  that 
will  show  the  impact  of  each  shipboard  equipment 
on  material  condition  readiness 

■  Provide  a  single  validation,  screening  and 
brokering  tool  for  use  across  all  ship  platforms 


Investment:  $0.5M 
Life  Cycle  ROI:  34:1 


■  Allow  for  a  near  term  predictive  nature  in  modeling 
accounting  for  failure  to  perform  routine 
maintenance 
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Intermittent  Fault  Detection  &  Isolation 


System 

Overview/Issue 

•  Unable  to  duplicate  discrepancy 
on  No  Fault  Found  (NFF)  LRU’s 

•  Bad  Actor  LRU’s  continued  to  be 
recycled  through  the  repair  cycle 

process 


Solution 

•  Develop  maintenance  tool  to 
augment  traditional  testing 
methods 

•  Will  identify  and  isolate 
intermittent  faults  on  end  items 

•  Repeats  Vigorous  Test  scenario 


(IFDIS) 


Investment/ROI 


Investment:  $2.20M 
Life  Cycle  ROI:  22:1 
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R-TOC  Project  Funding  vs.  Savings 


FY06-FY09  ($M) 


F  unding 

R  eliability 

Maintainability 

S  upportability 

Total 

Return  On 

Life  Cycle  S avings 

Life  Cycle  S avings 

Life  Cycle  S avings 

Life  Cycle  S avings 

Investment 

FY06 

23.077 

1,618 

943 

1,472 

4,032 

174:1 

FY07 

23.281 

12 

23 

208 

243 

10:1 

FY08 

25.598 

34 

519 

746 

1,298 

51:1 

FY09 

23.802 

725 

466 

998 

2,190 

92:1 

DoD  T otal 

95.758 

2,389 

1,951 

3,423 

7,763 

81:1 
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Initiatives  Contributing  to  R-TOC 


•  Lean  Enterprise  Value 

•  Six  Sigma 

•  Supply  Chain  Management 

•  DoD  Manufacturing  Technology  (ManTech) 

•  Value  Engineering 

-  FAR  provisions  offer  contractual  incentives 

-  Methodology  offers  approach  to  partner  with 
industry 
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Definition 


•  Value  Engineering  -  An  organized  effort  directed  at 
analyzing  the  functions  of  systems,  equipment, 
facilities,  services,  and  supplies  for  the  purpose  of 
achieving  the  essential  functions  at  the  lowest  life  cycle 
cost  consistent  with  required  performance,  reliability, 
quality,  and  safety.  OMB  Circular  A-1 31 

*  Bottom  Line:  Identify  and  Eliminate  Unnecessary  Cost 
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Value  Engineering  is  an 
R-TOC  Best  Practice 


•  VE  provides: 

-  Cost  reduction  (VEPs  and  VECPs) 

-  Product  or  process  improvement 

•  Higher  quality 

•  Reduced  cycle  time 

-  Better  means  and  materials  for  maintenance 

•  Increased  reliability 

•  Greater  safety 

•  Less  environmental  impact 

VE  Goal:  Lower  the  government’s  costs  for  goods  and 
services  &  provide  cost  effective  solutions  to  problems 
in  design,  development,  fielding,  support,  &  disposal 
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DoD  VE  Savings  &  Cost  Avoidance 


Fiscal  Year 
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VE  -  An  Industry  Example 


1998  Toyota  Corolla  -  VE  Project 

•  Problems:  Increased  material  costs,  production 
time  issues 


Objective:  Correct  problems  using  VE 


-  Lighter  by  10% 

-  25%  Fewer  engine  parts 

-  Faster  production 

-  Better  fuel  economy 

-  Decreased  emissions 

-  15%  Horsepower  increase 

-  Costs  $1,000  less  to  make  than  in  1997 
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VE  in  Systems  Engineering 


*  VE  methodology  is  an  effective  tool  for  making 
systems  engineering  decisions 

-  Reduce  cost 

-  Increase  productivity 

-  Improve  quality  related  features 

While... meeting  or  exceeding  functional  performance 
capabilities 

•  VE  is  applicable  at  any  point  in  the  life  cycle 

How... making  SE  trades 
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VE  and  R-TOC  in  Systems  Engineering 


*  VE  and  R-TOC  Early  in  the  Life  Cycle  -  Concept 
Refinement 

-  Analysis  of  Alternatives  -  evaluate  functions  vs. 
requirements 

-  Challenge  needs/ensure  requirements  are  valid 

-  SE  trades 

•  Develop  cost  of  alternatives 

-  Consider  life  cycle  cost  implications  -  (R-TOC) 


Savings  For  All  Production  Units 
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VE  and  R-TOC  in  Systems  Engineering 


*  VE  and  R-TOC  During  Technology  Development 

-  Analyze  value  of  requirements/specifications 

•  Can  these  be  tailored? 

-  Cost  as  an  independent  variable 

-  Compare  function,  cost  and  worth  of  technologies 

-  Consider  life  cycle  cost  implications  of  new 
technologies  -  R-TOC 
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VE  and  R-TOC  in  Systems  Engineering 


*  VE  and  R-TOC  During  Systems  Development  and 
Demonstration 

-  Identify  technical  approaches 

-  Eliminate  unnecessary  design  restrictions 

-  Estimate  cost  of  functions 

-  Identify  alternatives 

-  Evaluate  design  concepts  -  O&S  life  cycle 
concepts  (R-TOC) 

-  Search  for  new  technologies 

-  Simplify  designs 
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VE  and  R-TOC  in  Systems  Engineering 


•  VE  and  R-TOC  During  Production  and  Deployment 

-  Evaluate  and  improve  manufacturing  processes,  methods  and 
materials 

•  VE  and  R-TOC  During  Operations  and  Support 

-  Analyze  advances  in  technologies 

-  Evaluate  modifications 

-  Reduce  repair  costs 

-  Analyze  packaging  requirements 

-  Improve  RM&S  -  R-TOC 

-  Analyze/Improve  supply  chain/logistics  footprint  -  R-TOC 

-  Implement  CBM  -  R-TOC 

-  Reduce  manpower- R-TOC 
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SUMMARY 


•  R-TOC  and  VE  provide  savings/cost  avoidances  for  DoD 

•  VE  is  a  tool  for  Systems  Engineering 

•  R-TOC  provides  a  focus  on  life  cycle  design  considerations 

•  VE  supports  SE  trades 

•  Three  new  VE  documents:  1 )  VE  Contractor’s  Guide,  2) 
VECP  Contracting  Guide,  and  3)  VE  Handbook 

•  VE  revitalization  effort  in-work  -  USD(AT)  memo  on  OMB 
Circular  A-1 31 

•  R-TOC  is  driving  towards  institutionalization  of  O&S  cost 
reductions  across  all  programs 

•  R-TOCA/E  websites:  http://rtoc.ida.org  or  http://ve.ida.org 

•  R-TOC  /  VE  Points  of  Contact:  David  Erickson: 
David.Erickson@osd.mil  -  Danny  Reed:  dreed@ida.org 
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BACKUP 
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History 


•  VE  emerged  from  the  industrial  community. 

•  The  VE  concept  is  a  by-product  of  material 
shortages  during  WW  II  where  alternative 
approaches  often  worked  as  well  or  better  and 
cost  less. 

•  DoD  established  its  VE  Program  in  1963. 

•  VE  has  proven  to  be  a  successful  cost 
reduction/product  improvement  tool  for  over  40 

years! 
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VE  Application  Areas 


Examples:  How  VE  can  be  used: 

>  Improve/streamline  operations 

>  Improve  quality 

>  Increase  the  use  of  environmentally-sound  and 
energy-efficient  practices  and  materials 

>  Simplify  logistics 

>  Reduce  maintenance 

>  Increase  availability 

>  Improve  durability 

>  Reduce  cost 
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VE  Authority 


•  Office  of  Federal  Procurement  Policy  Act  41  USC 
432  -  Each  executive  agency  shall  establish  & 
maintain  cost-effective  VE  procedures  & 
processes 

*  Public  Law  Implemented  by  OMB  Circular  A-131 


•  All  Agencies  Will: 

Encourage  VECPs 
Encourage  VEPs 
Identify  and  report 
Train  in  VE 


-  Establish  and  maintain 
a  VE  Program 

-  Develop  annual  plans 

-  Budget  for  VE 


*  OMB  Circular  A-131  implemented  by  the  DoD  VE 
Strategic  Plan,  December  2003 
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Definition  of  Terms 


•  Value  Engineering  Change  Proposal  (VECP): 

A  proposal  submitted  by  a  contractor  that 
changes  the  contract  and  saves  the  government 
money 

•  Value  Engineering  Proposal  (VEP): 

A  government  generated  change  that  adds  value 
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VE  -  A  Systematic  Approach 


1.  Information  -  frame  the 
problem 

2.  Function  Analysis 

3.  Speculation  -  generate 
ideas  based  on  function 

4.  Evaluation  of  ideas 


5.  Development  of  ideas 

6.  Verification 

7.  Reporting  -  present 
business  case 

8.  Implementation  -  Use 
Champions  &  follow-up 


Provides  planned  systematic  approach  that  is  more 
productive  than  undisciplined  or  opportunistic  approach 
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Why  Optimization  May  Not  Have  Been 

Achieved 


*  Shortage  of  time 

*  Requirements  are  technically  beyond  capability 

*  Lack  of  insight  into  costs 

*  Change  in  technology  (hardware  &  processes) 

*  Lack  of  knowledge  of  actual  requirements 

*  Fixation  with  previous  designs 

*  Presence  of  bad  information/failure  to  fully 
communicate 

*  Habits  and  attitudes 

*  Temporary  circumstances 

*  Honest  but  wrong  beliefs 
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VE  Savings  Proposals 
In-house  (VEP)  Example 


Value  Engineering  Proposals 
(VEP)  - 

In  house  proposals  generate 
the  majority  of  savings  & 
cost  avoidances 


PA125  Ammo  Container 


Before 


After 


Govt  Savings  totaled 
$7i155  M  over  3  yea 


Sold  as  scrap 


Saved  for  refurbishment 
and  reuse  30 


DoD  VE  Strategic  Plan 


•  DoD  VE  Strategic  Plan  signed  by  USD(AT&L) 

-  Improve  value  for  defense  systems 

-  Align  industry  and  government  value 

-  Increase  VE  expertise 

•  Strategic  Plan 

-  Establishes  a  goal  of  VE  cost  savings  and  avoidances 
of  1 .5%  of  TOA  by  FY06  for  all  the  Services  and  other 
DoD  Agencies  -  Difficult  Stretch  Goal 

-  At  least  500  people  will  have  completed  VE  continuous 
learning  module  by  FY06  -  Currently  over  1100 

-  90%  of  VECPs  should  be  fully  processed  within  180 
days  by  FY06  -  Accomplished 

-  VECP  Community  of  Practice  (CoP)  -  Accomplished 
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Quantitative  Method  for  Project 

Evaluation 


•  Projects  submitted  for  R-TOC  funding  are  selected 
based  on  1 1  criteria,  five  objective  and  six  subjective 

•  Highest  values  are  placed  on  Return  on  Investment 
(ROI)  over  the  FYDP  and  over  the  Life  Cycle  of  the 
system  (using  OMB  discount  values)  and  on  a 
subjective  evaluation  of  improvements  in  Operational 
Readiness 

•  ROI  calculations  are  performed  automatically  by  use 
of  a  calculation  template 
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Lessons  Learned 


•  Most  Pilot  Programs  focused  on  “reduced  demand” 
via  Reliability,  Maintainability  and  Supportability 
(RM&S) 

•  A  few  programs  were  able  to  sign  long-term  contracts 
for  product  support  (“competitive  sourcing”),  which 
produced  O&S  savings  while  at  the  same  time 
increasing  readiness  and  availability 

•  Both  approaches  provided  savings  in  the  second 
listed  area  for  savings  -  reduced  response  times 
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Important  Design  Considerations 


Capabilities 


Functions 


Priorities 


System 

Performance 


Reliability 


System  performs nee 

Architectural 

impacts  on  System 

Open  Systems  Design 
Inter  operability 

Design 

Considerations 

ScrvivalbiNty  & 
Susceptibility 
software 
Anti-tamper 
HS1 

Access  bility 
[n$en&i0ve  Munitions 
System  security 
information  Assurance 
Corrosion  Prevention 
COTS 

O^pom  I  and 
Demilitarisation 
Environment,  Safety,, 
Occupational  Health 

Design  Tools 

^Modelnga  Simulation 


Mantainability  )- 


Scpportability  \- 


Ftoducbilicy  Y 


System 


Technical 

ETTecTi  ve  ne  is"'--.. 


Availability 


[  Operations  )- 


[  Maintenance 


|  Logistics 


Process 


System 

"Effectiveness 


Efficiency 


Life  Cycle  Cost  /  Total  Ownership  Cost 


Affordable 

Operational 

Effectiveness 


System  Availability 

Reliabi  luvMemta  inability  jEupdot  tabi  litv : 

RAM,  COTS 

Prodbcibiiity:  Val ue  Bngneering,,  Quality, 
Manufacturing  Capability 


Process  Efficiency 

Operations:  Interoperability,  Information  Assurance, 
Suitability  S  Susceptibility,  HSI,  System  Security,  Anti-tanrper 
Maintenance:  Corrosion  Prevention  end  Control,  Accessibility, 
Interoperability,  Unique  Identification  of  ]  tern? 

Lytles:  Suppar lability 

Rroduabilitv:  Value  Engnesring,  Quality,  Manufacturing 
Capability 
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Understanding  Systems  Engineering  by  Phase 


SE  Processes  (Engineering,  Supportability,  TA&E)  directly 
tied  to  technical  Inputs/Outputs  by  phase 

1  1  *  Outputs 


inputs 


•Sys  Performance  Spec 
•Exit  Criteria 

•Validated  Sys  Support  & 
Maintenance  Objectives  & 
Requirements 
•APB  •  CDD  •  SEP 
•  ISP  •  TEMP 


£ 


SDD 


•Test  Reports  •  TEMP 
•Initial  Prod  Baseline 
Elements  of  Product  Support 
•Risk  Assessment 
•SEP  *TRA  •  PESHE 
•Inputs  to: 

-CPD  -STA  -ISP 


Interpret  User  Needs, 
Refine  System 
Performance  Specs  & 
Environmental  Constraints 


FCA 

(svfQ  (j»rr) 

•••> 

Combined  DT &E/OT &E/LFT &E 
Demonstrate  System  to 
Specified  User  Needs  & 
Environmental  Constraints 

Develop  System 
Functional  Specs  & 
System  Verification  Plan 


Iterative 

and 

Recursive 


T&E  embedded  in  SE  process 


Evolve  Functional 
Performance  Specs  into 
Cl  Functional  (Design  to) 
Specs  and  Cl  Verification  Plan 


SE 


PDR 

Evolve  Cl  Functional 

Specs  into  Product 
(Build  to)  Documentation 


and  Insi 


Fabricate,  Assemble, 
Code  to  “Build-to” 
Documentation 


Technical  reviews  by  phase 


processes 
tailored  by 
phase 
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Technology  Development  Phase 


OUTPUTS 


INPUTS 


•ICD  &  Draft  CDD 
• Preferred  Sys  Concept 
•Exit  Criteria 
•T&E  Strategy 
•Support  &  Maintenance 
Concepts  &  Technologies 
• AoA  •  SEP  •  TDS 


•Sys  Performance  Spec 
•LFT&E  Waiver  Request 
•TEMP  •  SEP  • PESHE  •PPP  •TRA 
•Validated  Sys  Support  & 
Maintenance  Objectives  & 
Requirements 
•Footprint  Reduction 
•Inputs  to:  -  IBR  -ISP  -STA  -CDD 
-Acq  Strategy 
-Affordability  Assessment 
-Cost/Manpower  Est. 


Interpret  User  Needs. 

Csrr 

v  * 

Analyze  Operational 

^ — . _ _ 

Demo  &  Validate  Sys 

Capabilities  & 

Concepts  &  Technology 

Environmental  Constraints 

Maturity  Versus 

V^Trades^) 

Defined  User  Needs 

Develop  System  Perf 

tJL  Trades  il 

(&  Constraints)  Spec  & 

^ . 

. . ,v 

Demo/Model 

Enabling/Critical  Tech 
Verification  Plan 

A 

Integrated  System  Versus 
Performance  Spec 

Develop  Functional 
Definitions  for  Fnahlinn/ 

^ . iw 

Demo  System 

Uvl  II  lltlV/l  Iv  1  V/l  1—  1  1  Cl  bJ  1  1  i  11  Ml 

Critical  Technologies  & 
Associated  Verification  Plan 

t . t 

Functionality 

Versus  Plan 

Decompose  Functional 

Demo  Enabling/ 

Definitions  into  Critical 

<- . ■> 

Critical  Technology 

Component  Definition 

Components 

&  Tech  Verification  Plan 

Versus  Plan 

Develop  System  Concepts, 
i.e.,  Enabling/Critical  Technologies, 
Update  Constraints  & 
Cost/Risk  Drivers 
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Production  and  Deployment  Phase 


Independent  IOT&E 


Full-Up  System  Level  LFT&E 


JITC  Interoperability 
Certification  Testing 


J-6  Interoperability 
&  Supportability  Validation 


_ INPUTS _ 

'•  Test  Results  ' 

•Exit  Criteria 
•APB  •  CPD  •SEP 
•TEMP 

• Product  Support  Package  ^ 


OUTPUTS 


•Production  Baseline 
•Test  Reports 
•TEMP  •PESHE  • SEP 
•Input  to: 

-  Cost/Manpower  Est. 

V _  _ _ _ J 


PCA 

Analyze  Deficiencies 

> 

Verify  &  Validate 

D  I  I  I  /V  M 

1  o  ueiermine  uorrecuve 

Actions 

. 

Configuration 

Modify  Configuration 
(Hardware/Software/Specs) 
To  Correct  Deficiencies 
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Operations  and  Support  Phase 


INPUTS 


r  \ 

•Service  Use  Data 
•User  Feedback 
•Failure  Reports 
•Discrepancy  Reports 


V 


SEP 


J 


Monitor  and  Collect 
All  Service 
Use  Data 


Analyze  Data  to 
Determine 
Root  Cause 


Determine 
System  Risk / 
Hazard  Severity 


Integrate  &  Test 
Corrective  Action 


Develop 

Corrective 

Action 


•  Process  Change  - 
Hardware/Support 

•  Materiel  Change  38 


LifeCycle  Cost  Reduction  Potential 


COST  REDUCTION  POTENTIAL 


Concept 

Refinement 

Concept 

Decision 


Technology 

Development 

System  Development 
&  Demonstration 

*  Critical 
(y  Design 
v  Review 

Production  & 
Deployment 

A  FRP 

O  Decision 
v  Review 

Operations  & 
Support 
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USD(A&T)  FY  2005  R-TOC  Goal 


•  USD(AT&L)  Goal:  .reduce  the  O&S  cost  of  fielded 
systems  (excluding  manpower  and  fuel)  by  20% 
(compared  to  current  FY  1998  levels)  by  the  year  2005.” 

•  “Each  Military  Department  recently  proposed  ten  major 
‘pilot  program’  activities  to  test  Program  Manager 
performance...!  intend  to  use  the  pilot  programs  to 
demonstrate  the  type  of  cost  savings  depicted  in  the 
Defense  Planning  Guidance.” 

(Under  Secretary  of  Defense  Jacques  Gansler) 
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OSD  Funding  for 
Selected  R-TOC  Projects 


•  R-TOC  Program  Element  (PE)  was  created  using 
funds  ($25M  per  year)  provided  by  the  Services  in 
PBD  707 

•  PBD  707:  USD(AT&L)  and  the  Services  agreed  to 
move  1/3  each  from  the  Services  O&M  accounts  to 
an  OSD  account  to  fund  the  R-TOC  PE 

•  Competitive  funding  of  projects  submitted  by 
Services  to  reduce  long-term  O&S  costs 
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SAFE-SEPARATION  ANALYSIS 
SYSTEM  SAFETY  ENGINEERING  STUDY 


Ken  Chirkis,  Jason  Cushing,  Steve  Bussell 
NAWCWD  China  Lake 

Dave  Hall,  Ray  Terry 
SURVICE  Engineering 


Cleared  for  Public  Release 
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Background 


•  Tasking  from  PMA-201  to  review  safe  escape  -  safe  separation  analysis 
methodologies  across  the  Services 

-  NAWCWD  China  Lake:  Ken  Chirkis,  Jason  Cushing,  Steve  Bussell 

-  SURVICE  Engineering:  Dave  Hall,  Ray  Terry,  Mike  Ray 


•  Presented  results  to  the  DoD  Fuze  Engineering  Standardization 
Working  Group  (FESWG),  28  -  30  November  2006 

-  FESWG  recommended  we  brief  the  DOD  Fuze  IPT 

-  Study  recommends  changes  to  Joint  definitions,  establishing  Joint 
guidance  for  analysis  assumptions  and  methodologies 


•  Presented  results  to  DOD  Fuze  IPT,  28  February  2007 

-  IPT  supported  recommendations  for  changes  to  standards  and 
process  documents 

-  Recommended  FESWG  as  the  technical  standards  group 


Presented  results  to  Fuze  Safety  Summit,  March  2007 
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Statement  of  Work 


*  Review  Safe  Separation/Safe  Escape  Analysis  Approaches 

-  Analyses  to  determine  minimum  arm  time/distance,  safe 
escape  release  conditions,  risk  assessments  for  air 
launched  weapons  systems 

-  Examine  requirements,  approaches,  assumptions, 
methodologies 

-  Account  for  post-release  aircraft  maneuvering 

*  Compare  service  approaches 

*  Consider  additional  sources  of  information 

-  Survivability  analyses  (aircraft  vulnerability  models) 

-  Other  known  risks  to  aircraft  (enemy  weapons,  etc.) 

*  Provide  independent  recommendations  for  improvement 

*  Prepare  briefing  on  results  to  PMA-201  Fuze  IPT  System  Safety 
Working  Group  (SSWG) 
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Technical  Approach 


•  Develop  Consistent  Evaluation  Questionnaire 

-  Assumptions  (post-launch  aircraft  maneuvers,  weapon 
variations,  environmental  variations,  launch  modes,  S/A  device 
variations) 

-  Requirements  (risk  probability,  hit  and/or  kill,  analysis 
objectives,  post-launch  maneuver  requirement) 

-  Definitions  (safe  separation,  safe  escape,  safe  arming,  definition 
source) 

-  Aircraft  Modeling  (flight  path,  physical  description,  vulnerability, 
maneuvers,  air  target  maneuvers) 

-  Weapon  Modeling  (trajectory,  debris  model  fidelity,  variations, 
S/A  device  modeling) 

-  M&S  and  Credibility  (what  M&S,  capability,  accuracy,  usability) 

•  Interview  Service  Safe-Separation/Safe-Arming  Analysts 

•  Analyze  Interview  Results  (and  any  additional  data  collected) 

•  Formulate  Recommendations 

•  Document  Results 
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Data  Collection  Results 


•  Interviewed  NAWCWD  and  AMRDEC  analysts 

-  NAWCWD  Warfare  Analysis  Division  at  China  Lake 

-  Aviation  Engineering  Directorate  at  Redstone  Arsenal 

•  Seek  Eagle  and  NAWCAD  analysts  declined  to  be  interviewed 

-  Referred  us  to  JSF  JSEAS  effort 

•  Joint  Safe  Escape  Analysis  Solution 

-  JSF  provided  document  “JSF  Common  Safe  Escape  Criteria” 

•  Agreement  on  23  joint  requirements  for  safe  escape  analysis 

•  Covers  air-to-ground  weapons 

•  Participants  included  NAWCAD,  Seek  Eagle,  UK  analysts 

•  Covered  most  of  our  interest  in  requirements,  very  little  in  other 
categories  of  information  (assumptions,  M&S,  etc.) 

•  We  filled  in  some  information  from  other  sources 

-  NAWCWD  and  Seek  Eagle  have  close  working  relationship 

-  Air  Force,  Army  briefings  from  April  06  Seek  Eagle  conference 
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Results: 

Assumptions 


Assumption 

NAWCWD 

NAWCAD 

SEEK  EAGLE 

AMRDEC 

Launch 

aircraft 

maneuvers 

Assume  straight  and 
level  is  worst  case; 
fixed  “g” 
maneuvers; 
altitudes  &  speeds 
from  tactics  guides 

Assume  straight 
and  level  is  worst 
case;  fixed  “g” 
maneuvers; 
altitudes  &  speeds 
from  tactics  guides 

Hover,  Bank, Dive, 
attack  run,  break 
turn  toward 
masking  terrain 
after  launch,  or 
vertical  or  lateral 
unmask  &  egress 

Weapon 

Variations 

Hot/cold  motor  when 
data  available;  no 
roll  variations; 
variable  launch 
modes 

Hot/cold  motor 
when  data 
available;  no  roll 
variations;  variable 
launch  modes 

Hot/cold  motor 
when  data 
available  and  IFS 
of  sufficient  fidelity 

S/A  Device 
Variations 

Spec  value  plus  and 
minus  tolerance 

Spec 

value 

minus 

tolerance 

Spec  value  minus 
tolerance  +  delay 

UNK 
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Results: 

Requirements 


Requirement 

NAWCWD 

NAWCAD 

SEEK  EAGLE 

AMRDEC 

Launcher 

vulnerability 

metric 

Hit  &  Kill 

Hit  Only 

Hit  Only 

Hit  (frag  KE>5 
ft-lbs  or  V>V50) 

&  Kill 

Probability 

requirement 

.0001  or  .01*  or 
outside  hazards 
analysis 

.0001 

.0001  or  .01*  or 
outside  hazards 
analysis 

Zero,  or  1 0  6 

In  some  cases 
may  use  .0001** 

Maneuver  after 
launch  required 
if  probability 
not  met? 

Yes  (in  one  or 
two  cases) 

No 

Analysis 

Objectives 

Safety  of  flight 
clearance;  safe 
escape 

maneuver 

determination 

Safety  of  flight 
clearance;  safe 
escape 

maneuver 

determination 

Minimum  low- 
altitude  safe 
release  range; 
risk 

assessment 

*  Modified  by  Pdet 

**  AMRDEC  System  Simulation  and  Development  Directorate 
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Army  Hazard  Matrix 


HAZARD  MATRIX 


Catastrophic 


Could  result  in  death,  permanent  total 
disability,  loss  exceeding  $1M,  or 
irreversible  severe  environmental 
damage  that  violates  law  or  regulation.  I 


Critical  o 

Could  result  in  permanent  partial 
disability,  injuries  or  occupational 
illness  that  may  result  in  hospitalization 
of  at  least  three  personnel,  loss 
exceeding  $200K  but  less  than  $1M, 
or  reversible  environmental  damage 
causing  a  violation  of  law  or  regulation. 


Marginal 


Could  result  in  injury  or  occupational 
illness  resulting  in  one  or  more  lost 
work  days(s),  loss  exceeding  $10K  but 
less  than  $200K,  or  mitigatible 
environmental  damage  without  violation 
of  law  or  regulation  where  restoration 
activities  can  be  accomplished. 


Negligible 


Could  result  in  injury  or  illness  not 
resulting  in  a  lost  work  day,  loss 
exceeding  $2K  but  less  than  $10K,  or 
minimal  environmental  damage  not 
violating  law  or  regulation. 


Frequent  ^ 

Likely  to  occur  often 
in  the  life  of  an  item, 
with  a  probability  of 
occurrence  greater 
than  0.1  in  that  life. 


Probable  p 

Will  occur  several  ^ 
times  in  the  life  of  an 
item,  with  a  probability 
of  occurrence  less  than 
0.1  but  greater  than 
0.01  in  that  life. 


Occasional  q 

Likely  to  occur  some 
time  in  the  life  of  an 
item,  with  a  probability 
of  occurrence  less  than 
0.01  but  greater  than 
0.001  in  that  life. 


Remote  q 

Unlikely  but  possible  to 
occur  in  the  life  of  an 
item,  with  a  probability 
of  occurrence  less 
than  0.001  but  greater 
than  10*6  in  that  life. 


Improbable  £ 

So  unlikely,  it  can  be 
assumed  occurrence 
may  not  be  experience^ 
with  a  probability  of 
occurrence  less  than 
lO*6  in  that  life. 


Drive  S/A 
Results  to 
Blue  Area 
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From:  Fuze  Management  Board 
Joint  Agreement  (1978) 


•  Pkill:  “If  the  minimum  safe-separation  distance  (resulting  from  the 
Phit<.0001  requirement)  restricts  tactical  delivery  conditions,  the 
probability  of  a  fragment  hit  may  be  further  qualified  by  considering  only 
the  presented  area  of  critical  systems  or  components  rather  than  the 
area  of  the  complete  launching  system.” 

-  Interpreted  by  NAWCWD  (and  AMRDEC)  as  Pkill 

-  UK  uses  “self  damage”  metric 

•  Risk  Analysis:  “If  the  above  procedures  (Phit  or  Pkill  <.0001)  still  result 
in  restricting  tactical  delivery  conditions,  then  selected  fuze  arming 
conditions  which  are  such  that  a  safe-separation  distance  is  not 
achieved  must  be  justified  by  a  thorough  analysis.” 

-  “This  analysis  should  consider  probability  of  a  specific  type  of 
damage,  decreased  risk  from  enemy  ordnance,  and  tactical 
advantage  gained  by  use  of  the  recommended  fuze  arming 
characteristics” 

•  Fragment  Hit:  “A  fragment  which  contains  sufficient  kinetic  energy  to 
penetrate  the  launch  aircraft  skin  which  is  exposed  to  the  hazard.” 

-  Army  uses  KE>5  ft-lbs,  or  V50  analysis 

-  Not  clear  what,  if  anything,  anyone  else  uses  as  hit  criteria 
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Safe  Escape  Analysis  Requirements 


(From  NAWCWD 
Briefing) 


Note:  Pdet  cannot  be 
less  than  .01 
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Phjt  Requirement 
and  Historical  Data 


•  KH  requirement  purported  to  be  based  on  historical  data 

*  No  documentation  available  from  original  decision  (1978) 

•  Analyzed  available  hit  rate  data  from  SEA  and  Desert  Storm 

*  Obtained  mishap  rate  data  for  F-16  and  UAV  systems 


Compared  to  Phit  requirement 
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SEA  Hit  Rate  Experience 
Air  Force  Aircraft 


4000 1 

Y  , 

3500 

3000 

2500 

2000 

i 

7 

1500 

1000 

500 

/  m  y  y\ 

0 

z 

C-130  C-123  F-4  AC-130 


□  Sorties  per  hit 


Phi,  per  sortie  ~  1 0'3  for  transports,  1 0'2  for  attack  a/c 


Source:  ASD/XRM  Analysis 
SURVIAC  Data 
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SEA  Hit  &  Kill  Rates 


USN  &  USMC  Fixed  Wing  Aircraft, 
(Apr  1965 -Mar  1973) 


Service 

Hit  Rate 
(per  1000 
sorties) 

Kill  Rate 
(per  1000 
sorties) 

USN 

5.23 

1.05 

USMC 

6.32 

0.54 

Phit 

per  sortie  ~ 

10-2 

Pkill 

per  sortie  ~ 

10-3 

Source:  U.S.  Navy,  Marine  Corps  and  Air  Force  Fixed  Wing  Aircraft  Losses  and  Damage  in  Southeast 
Asia  (1962-1973),  Center  for  Naval  Analyses,  Aug  1976 
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Desert  Storm 
Hits  by  Mission  Type 


□  Sorties  (XI 000) 
n  Hit  Incidents 


Interdiction  DCA  CAS  SEAD 


Phit  per  sortie  ~  1 0-3 

nlt  Source:  SURVIAC 

(or  zero,  for  DCA  -  no  threat  a/c) 
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Mishap  Rate  Comparison 


Mishap  Rate  Approaches  10~4  as  cumulative  flight  hours  approach  100,000 


Source:  UAS  Roadmap  2005 
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Phjt  Requirement 
and  Historical  Data 


•  Historical  Data  Summary 

-  SEA  and  Desert  Storm  Combat  hit  rates  per  sortie  vary  from  10~2 
to  10-3,  depending  on  aircraft  type  and  mission 

-  Aircraft  combined  Class  A  and  B  mishap  rates  per  flight  hour 
converge  to  around  10-4 

•  Apples  and  Oranges: 

-  Mishap  rate  per  flight  hour 

-  Combat  hit  rate  per  sortie 

-  Weapon  fragment  hit  probability  per  weapon  release 

•  However,  a  10~4  requirement  is  not  inconsistent  with  overall 
historical  rates 

-  Not  exactly  supported  by  history,  but  not  completely  out  of  line 

-  Combat  hit  rates  support  “additional  analysis  of  other  risks”  to 

justify  not  meeting  probability  requirement  @ 
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MIL-HDBK-1763  Definitions 


3.1.27  Safe  escape/safe  arming 

Safe  escape  is  the  minimum  release  altitude  which  will  provide  the  delivery 
aircraft  acceptable  protection  from  weapon  fragmentation  for  detonation  at 
the  preplanned  point.  Safe  arming  separation  is  the  selection  of  a  minimum 
safe  fuze  arm  time  setting  which  will  provide  the  delivery  aircraft  acceptable 
protection  from  weapon  fragmentation  if  early  detonation  should  occur. 


3.1.28  Separation 

The  terminating  of  all  physical  contact  between  a  store,  or  portions  thereof, 
and  an  aircraft;  or  between  a  store,  or  portions  thereof;  and  suspensions 
equipment. 


3.1.28.1  Safe  separation 

The  parting  of  a  store(s)  from  an  aircraft  without  exceeding  the  design 
limits  of  the  store  or  the  aircraft  or  anything  carried  thereon,  and  without 
damage  to,  contact  with,  or  unacceptable  adverse  effects  on  the  aircraft, 
suspension  equipment,  or  other  store(s)  both  released  and  unreleased. 


3.1.28.2  Acceptable  separation 

Acceptable  store  separations  are  those  which  meet  not  only  the  "safe" 
separation  criteria,  but  also  meet  pertinent  operational  criteria.  For 

instance,  guided  weapons  as  a  minimum  must  remain  within  control 
limitations  consistent  with  mission  effectiveness.  Conventional  weapons, 
bombs,  should  not  experience  excessive  angular  excursion  which  induce 
ballistic  dispersions  that  adversely  affect  weapons  effectiveness,  or  bomb- 
to-bomb  collisions. 


Other  Documents’  Definitions 
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MIL-STD-1316E: 

Safe  Separation  Distance:  The  minimum  distance  between 
the  delivery  system  (or  launcher)  and  the  launched 
munition  beyond  which  the  hazards  to  the  delivery  system 
and  its  personnel  resulting  from  the  functioning  of  the 
munition  are  acceptable. 

1978  Joint  Agreement: 

Safe-Separation  Distance:  the  minimum  distance  between 
the  launching  system  (AIRCRAFT  &  PILOT)  and  its 
launched  munitions  at  which  hazards  associated  with 
munitions  functioning  are  acceptable.  This  distance  may 
be  achieved  by  providing  arming  delays(s)  (time  or 
distance). 
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Analysis  Definitions 


•  All  analysts  in  all  Services  call  what  they  do  “safe  escape 
analysis”  vice  “safe  separation  analysis” 

-  Consequently,  should  consider  changing  the  MIL-STD  and 
Joint  Agreement  definitions  to  make  “safe  separation”  mean 
safe  release  of  the  weapon  from  the  launch  mechanism 

-  Change  “safe  separation  distance”  to  “safe  arming  distance 
or  “safe  escape  distance” 

•  However,  not  all  safe  escape  analyses  involve  determining 
minimum  release  altitude  (MRA)  or  minimum  safe  release 
altitude  for  fragment  avoidance  (MinAlt)  per  the  MIL-HDBK 
definition 

-  Air  to  air  analyses  do  not  in  general  determine  safe  release 
altitudes 

•  So  there  is  still  some  difference  in  definition  of  safe  escape 
analysis  that  needs  to  be  resolved 
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Results: 

Aircraft  Modeling 


Aircraft 

Modeling 

Issue 

NAWCWD 

NAWCAD 

SEEK  EAGLE 

AMRDEC 

Physical 

Description 

6-view  presented 
area 

6-view  presented 
area 

6-sided  box  enclosing 
aircraft  +  CAD  model 

Vulnerability 

Description 

6-view  vulnerable 
area  (from 
survivability 
analysis) 

NA 

NA 

AJEM  model 

Target 

Maneuvers 

(air-to-air) 

Straight  and  level 
(assumed  worst 
case);  occasionally 
consider  target 
maneuvers 

UNK 

NA 

Aircraft 

Flight  Path 
Model 

JAAM 

JAAM,  AWDS 

RCAS  or  FlightLab 

Target 

Debris 

Model 

Not  modeled 

Not  Modeled 

Not  Modeled 
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Results: 

Weapon  Modeling 


Weapon 
Modeling  Issue 


NAWCWD 


NAWCAD 


SEEK  EAGLE 


AMRDEC 


Weapon 

trajectory 


Program  office  6-dof 


Program  office  6- 
dof 


Program  Office 
6-dof 


Motor 

Temperature 


Hot/Cold  variations  if 
data  available 


Hot/Cold  variations 
if  data  available 


Hot/Cold 
variations  if  data 
available 


Debris  source 


Arena  Test  Data 


Arena  Test  Data 


Arena  Test  Data 


Debris  frag 
zones 


5-10  deg  polar  zones; 
uniform  distribution 


10  deg  polar  zones; 
24  roll  zones 


5  deg  polar 
zones;  unif.  dist. 


Debris  Frags 


Large  frags  & 
warhead  frags 
modeled  separately; 
no  min  frag  size  or 
velocity;  no  data 
available  for 
statistical  variations 


Large  frags  & 
warhead  frags 
modeled 

separately;  no  data 
for  statistical 
variations;  unk  min 
frag  size  or  velocity 


UNK  treatment  of 
large  &  warhead 
frags;  small  frags 
KE<5  ft-lbs 
removed;  Monte 
Carlo  frag  flyout 
simulation 


S/A  Device 


Arm  time  plus  & 
minus  spec  tolerance 


Spec  value 

minus 

tolerance 


Spec  value  minus 
tolerance  +  delay 


UNK 
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Results: 

M&S  &  Credibility 


M&S  Issue 

NAWCWD 

NAWCAD 

SEEK  EAGLE 

AMRDEC 

M&S  Used 

ASEP 

Path  4 

CASES 

ASEAT 

Capability 

Adds  asymmetric 
roll  zones  to  Path  3D 

3D 

dynamic 
frag  zones 

Pre-generated 
warhead  data  files; 
adds  GUI  to  ASEP 

Monte-Carlo, 
two  passes  (hit 
box,  then  CAD 
model) 

Accuracy 

No  formal  V&V; 
comparison  runs 
between  ASEP  & 
CASES;  no  data  V&V 
documented;  no 
formal  validation; 
accreditation 
package  done  by 
SEEK  EAGLE 

No  formal  V&V; 
comparison  runs 
between  ASEP  & 
CASES;  no  data 

V&V  documented; 
no  formal 
validation; 
accreditation 
package  done  by 
SEEK  EAGLE 

AJEM  V&V;  no 
V&V  or 

documentation 
available  on 
ASEAT  and 
associated 

M&S 

Usability 

User  Manual  & 
Analyst  Manual; 

SEEK  EAGLE 
provides  limited 
user  support 

UNK 

Documentation; 

SEEK  EAGLE 
provides  user 
support 

Recommendations 
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•  Assumptions:  Should  be  Joint  guidance  for  assumptions 
used  in  safe-escape  analyses 

-  Launch  aircraft  maneuvers,  weapon  variations  (angle  of 
attack,  motor  temperature,  roll  orientation,  etc.), 
environmental  factors,  safe-arm  device  variations,  and 
other  factors  that  potentially  drive  the  analysis  results 

•  Requirements:  JSEAS  requirements  should  serve  as  the 
starting  point  for  expansion  to  include  Army  requirements 
and  air-to-air  weapon  system  requirements 

-  Include  provision  for  application  of  the  process  outlined 
in  the  original  Joint  Agreement  between  all  the  Services, 
particularly: 

•  Inclusion  of  PkiN  as  a  metric 

•  Provision  for  additional  analyses  to  support  operational  use  of 
weapons  that  do  not  meet  the  0.0001  probability  requirement 
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Recommendations  (Continued) 


•  Recommend  changing  MIL-HDBK-1763  Definitions: 

-  Safe  escape:  Safe  escape  is  the  required  release  conditions 
and  post-launch  maneuvers  that  will  provide  the  delivery 
aircraft  acceptable  protection  from  weapon  fragmentation  for 
detonation  at  the  preplanned  point  or  at  or  after  arming;  this 
may  result  in  a  minimum  safe  release  altitude. 

-  Safe  arming:  Safe  arming  is  the  selection  of  a  minimum  safe 
fuze  arm  setting  that  will  provide  the  delivery  aircraft 
acceptable  protection  from  weapon  fragmentation  if 
detonation  should  occur  at  or  after  the  fuze  arm  time/distance. 

•  Also  change  MIL-STD-1316E  and  Fuze  Management  Board  Joint 
Agreement  definitions  of  “safe  separation  distance”  to  be  “safe 
arming  distance”  (or  “safe  escape  distance”) 

•  Would  require  fairly  extensive  changes  to  MIL-HDBK-504 
processes  and  definitions 
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Recommendations  (Continued) 


•  Aircraft  Modeling:  Should  be  guidance  for  launch  aircraft  post¬ 
launch  maneuvers  to  consider  for  safety  reasons. 

-  Conduct  Sensitivity  Analyses  to  determine  whether  there  is  a 
need  for  more  detailed  aircraft  representations  than  6-view 
presented  areas  (as  in  AMRDEC  approach) 

*  Weapon  Modeling:  Should  be  guidance  for: 

-  Fidelity  of  weapon  debris  modeling  (polar  zones,  etc.). 

-  When  to  segregate  “unusual”  fragments  for  separate  analysis 

•  Such  as  bomb  lugs,  warhead  fragments  that  are  likely  to  have 
much  higher  velocities  than  debris  fragments,  etc. 

-  What  fragments  to  include  in  the  weapon  debris  model 

•  Capable  of  penetrating  the  skin  of  the  aircraft 

-  Per  the  Joint  Agreement  definition  of  “fragment  hit” 

•  Consistent  with  the  Army’s  KE>5  ft-lbs  requirement  for  fragment 
inclusion  in  the  debris  model  (or  V50  analysis) 

-  Conduct  sensitivity  analyses  to  determine  requirement  for 
variations  in  weapon  orientation  (roll,  pitch, yaw)  and  effect  on 
results 
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Recommendations  (Continued) 


•  M&S  and  Credibility: 

-  USN  representatives  should  consider  migrating  to  the 
latest  version  of  the  Seek  Eagle  methodology  (CASES) 

-  When  available,  the  JSEAS  methodology  should  be 
assessed  for  adoption  as  the  standard  Joint  Service 
methodology 

-  Documented  verification  and  validation  evidence  should 
be  developed  for  all  M&S  tools  used  in  safe  escape/safe 
arming  analyses 

-  Documentation  of  all  methodologies  used  by  the 
Services  should  be  developed,  maintained  and 
distributed  to  users 

-  An  Accreditation  Support  Package  (ASP)  should  be 
developed  for  the  M&S  tools  that  are  continuing  in  use 


Status 
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•  Prepared  a  draft  revision  of  the  1978  Joint  Fuze  Management  Board 
Agreement  on  safe-escape  analyses 

-  Will  present  draft  to  the  FESWG  for  review  and  action 

•  Developed  draft  revisions  of  definitions  and  methodology 
descriptions  in: 

-  MIL  HDBK  1763,  Aircraft/Stores  Compatibility:  Systems 
Engineering  Data  Requirements  And  Test  Procedures 

-  MIL  STD  1316E  &  F,  Department  of  Defense  Safety  Criteria  For 
Design  Criteria  Standard,  Fuze  Design 

-  MIL  HDBK  504,  Guidance  On  Safety  Criteria  For  Initiation 
Systems 

-  STAN  AG  4187E4,  Fuzing  Systems  Safety  Design  Requirements 

-  MIL  STD  191 1  A,  Department  Of  Defense  Design  Criteria 
Standard,  Safety  Criteria  For  Hand-emplaced  Ordnance  Design 
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ABSTRACT 


The  Naval  Air  Systems  Command  (NAVAIR)  contracted  with  SURVICE  Engineering 
Company  to  review  current  technical  requirements,  approaches,  assumptions  and 
methodologies  associated  with  the  determination  of  safe-separation  (minimum  arm-time 
or  arm  distance)  and  safe-escape  (weapons  target  impact)  calculations  and  corresponding 
release  conditions  for  air  launched  weapons  systems.  This  document  reports  the  results 
of  that  study,  comparing  two  Navy  approaches:  one  at  the  Naval  Air  Warfare  Center, 
Weapons  Division  (NAWCWD),  China  Lake,  CA,  and  the  other  at  the  Naval  Air  Warfare 
Center  Aircraft  Division  (NAWCAD),  Patuxent  River,  MD;  the  Air  Force  Seek  Eagle 
Office  approach  at  Eglin  AFB,  FL;  and  the  Army  approaches  at  the  Aviation  Engineering 
Directorate  at  Redstone  Arsenal,  AL.  The  study  team  conducted  interviews  with 
available  service  experts;  reviewed  briefings  and  papers  presented  in  various  venues;  and 
analyzed  available  modeling  and  simulation  (M&S)  documentation.  The  study  also  drew 
on  the  results  of  the  ongoing  Joint  Strike  Fighter  (JSF)  effort  to  develop  a  Joint  Safe 
Escape  Analysis  Solution  (JSEAS).  The  comparison  criteria  included  assumptions, 
requirements,  definitions,  aircraft  modeling,  weapon  modeling,  and  the  safe  escape/safe 
arming  modeling  and  simulation  suites  used  by  the  various  service  commands.  The  study 
concluded  with  recommendations  for  improvement  in  each  of  those  areas. 
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PREFACE 

This  work  was  conducted  for  the  Naval  Air  Warfare  Center  Weapons  Division  at  China 
Lake,  CA,  Systems  Safety  Division.  The  technical  monitor  was  Mr.  Ken  Chirkis.  The 
SURVICE  Engineering  Lead  Analyst  for  the  project  was  David  Hall,  working  from  the 
Ridgecrest  Area  Operation,  900E  N.  Heritage  Drive  Suite  1,  Ridgecrest,  CA,  93555, 
(760)  446-2424,  dave.hall@survice.com. 


3 


INTRODUCTION 


Safe  Separation  analyses  (also  called  Safe  Arm  analyses  and  Safe  Escape  Analyses)  are 
conducted  as  part  of  the  system  safety  program  for  air  weapons  systems.  These  analyses 
are  conducted  for  a  variety  of  related  purposes:  to  develop  safe-arming  times  (or 
distances)  to  be  designed  into  the  weapon’s  safety  and  arming  device;  for  a  risk 
assessment  as  part  of  a  safety  of  flight  analysis;  to  determine  safe  escape  maneuvers  that 
may  be  required  of  the  launch  aircraft  in  order  to  meet  safety  requirements;  and  to 
support  safety  of  flight  clearance  for  the  weapon. 

Currently,  the  Services  use  somewhat  different  approaches  to  determine  safe-escape 
requirements  and  weapons  release  conditions.  These  analyses  are  conducted  at  the  Naval 
Air  Warfare  Center  Aircraft  Division  (NAWCAD),  Naval  Air  Warfare  Center  Weapons 
Division  (NAWCWD),  the  Seek  Eagle  office  at  Eglin  Air  Force  Base,  the  Aviation 
Engineering  Directorate  of  the  Aviation  &  Missile  Research,  Development  & 
Engineering  Center  (AMRDEC)  at  Redstone  Arsenal,  and  the  AMRDEC  System 
Simulation  and  Development  Directorate,  Endgame  Analysis  Branch.  Past  comparisons 
between  the  results  of  these  analyses  for  Joint  weapons  systems  have  shown  that  in  some 
cases  the  results  from  one  Service  activity  derive  release  conditions  that  are  restrictive  or 
result  in  specific  weapons  capabilities  not  being  authorized  for  use,  whereas  analysis  by 
another  Service  activity  may  obtain  a  different  result. 

While  NAWCAD  and  NAWCWD,  for  example,  have  specific  commodities  for  which 
they  are  each  responsible  (powered  weapons  at  NAWCWD  and  gravity  weapons  at 
NAWCAD),  it  is  reasonable  to  expect  approaches  to  be  relatively  consistent  between  the 
two  organizations.  In  the  past,  attempts  have  been  made  to  understand  the  similarities 
and  differences,  but  this  effort  has  not  been  pursued  to  conclusion.  With  more  Joint 
weapons  entering  service,  it  is  imperative  that  release  conditions  be  consistent  among  the 
services  and  that  they  provide  effective  employment  tactics  while  maintaining  adequate 
safety  margins. 

As  a  result  of  this  issue,  the  Naval  Air  Systems  Command  (NAVAIR)  contracted  with 
SURVICE  Engineering  Company  to  review  current  technical  requirements,  approaches, 
assumptions  and  methodologies  associated  with  the  determination  of  safe-separation 
(minimum  arm-time  or  arm  distance)  and  safe-escape  (weapons  target  impact) 
calculations  and  corresponding  release  conditions  and  to  provide  independent 
recommendations  for  improvement  of  this  capability.  This  task  was  coordinated  with  the 
Weapon  System  Explosives  Safety  Review  Board  (WSESRB)  and  its  associated  sub¬ 
panels  such  as  the  Fuze  and  Initiation  System  Technical  Review  Panel  (FISTRP)  and 
Software  System  Safety  Technical  Review  Panel  (SSSTRP).  SURVICE  personnel 
attended  the  PMA-201  Fuze  Integrated  Product  Team  (IPT)  System  Safety  Working 
Group  (SSWG),  as  part  of  the  system  safety  engineering  support  to  program  activities 
relating  to  requirements  of  the  WSESRB.  The  results  of  this  project  were  briefed  to  the 
Fuze  Explosives  Safety  Working  Group  (FESWG)  and  the  DOD  Fuze  IPT. 
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APPROACH 


SURVICE  developed  a  study  plan  that  was  approved  by  the  Technical  Monitor.  The 
elements  of  that  study  plan  were  as  follows: 

1.  Develop  Consistent  Comparison  Criteria:  SURVICE  developed  a  matrix  of 
comparison  criteria  to  use  for  conducting  a  detailed  survey  of  the  various 
approaches  used  at  NAWCWD,  China  Lake  (for  Navy  and  Marine  Corps  air 
launched  powered  weapons  systems),  NAWCAD,  Patuxent  River  (for  Navy  and 
Marine  Corps  air-launched  gravity  weapons),  at  the  Seek  Eagle  office  at  Eglin 
AFB  for  Air  Force  weapons,  and  at  the  Aviation  Engineering  Directorate  of 
AMRDEC  at  Redstone  Arsenal  for  Army  air  launched  weapons.  These  criteria 
include  methodology,  data,  and  process  issues  (Appendix  A). 

2.  Interview  Service  Safe-Separation/Safe-Arming  Analysts:  SURVICE  planned 
to  meet  with  personnel  involved  in  safe-separation  and  safe-escape  analyses  at 
each  of  the  organizations  to  review  all  aspects  of  their  methodology  for 
performing  safe-escape  and  safe-separation  calculations  against  the  comparison 
criteria.  Of  particular  interest  was  whether  any  of  the  models  used  are  formally 
accredited.  This  approach  had  to  be  modified  when  Air  Force  and  NAWCAD 
personnel  were  not  available  to  be  interviewed.  Where  service  personnel  were 
not  available  for  interview,  SURVICE  attempted  to  fill  in  their  information  by 
interviewing  other  available  experts  in  the  field  and  reviewing  previous  briefings 
and  reports.  In  addition,  NAWCAD  and  Air  Force  personnel  directed  SURVICE 
to  an  ongoing  project  for  the  Joint  Strike  Fighter  (JSF)  program  to  develop  a 
“Joint  Safe  Escape  Analysis  Solution  (JSEAS)”  for  JSF. 

3.  Analysis  of  Interview  Results:  SURVICE  compared  the  various  Service 
methodologies  using  the  comparison  criteria.  This  analysis  used  the  results  of 
task  2  plus  all  additional  available  information.  The  outcome  of  this  task  was  a 
draft  of  the  main  body  of  this  final  report,  highlighting  differences  in  each  of  the 
methodologies. 

4.  Recommendations:  SURVICE  developed  some  recommendations  regarding  best 
practices  for  safe  separation/safe  arming  analyses.  These  recommendations  take 
into  account  the  ongoing  JSF  JSEAS  development,  and  include  methodologies, 
data,  processes,  acceptable  levels  of  risk,  and  standardized  terminology. 

5.  Documentation:  The  study  results  were  documented  in  this  final  report,  as  well 
as  in  a  briefing  prepared  for  the  PMA-201  Fuze  IPT  System  Safety  Working 
Group,  the  Fuze  Explosives  Safety  Working  Group  (FESWG)  and  the  DOD  Fuze 
IPT. 
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DISCUSSION 


The  first  task  in  this  project  was  to  develop  a  set  of  consistent  criteria  against  which  to 
compare  the  various  approaches  taken  by  the  service  facilities  to  conducting  safe- 
separation/safe-arming  analyses.  These  criteria  were  expressed  in  the  form  of  interview 
questions.  Appendix  A  lists  the  set  of  questions  that  were  asked  of  each  of  the  service 
experts;  these  questions  had  to  do  with: 

1.  Assumptions:  assumptions  made  regarding  launch  aircraft  maneuvers,  both 
before  and  after  weapon  release;  variations  in  weapon  trajectory;  environmental 
variations;  launch  modes;  variations  in  safety  and  arming  device  functioning 

2.  Requirements:  requirements  used  to  evaluate  weapon  system  safety  (probability 
of  hit,  probability  of  kill,  what  values);  whether  post-launch  maneuvers  may  be 
required  to  meet  safety  requirements;  objectives  of  the  analyses  (risk  assessment 
only,  determining  safe-arm  time/distance,  safe  escape  maneuver,  safety  of  flight 
clearance) 

3.  Definitions:  terms  used  to  describe  the  analyses,  such  as  safe-separation  analysis, 
safe  escape  analysis,  safe  arming  analysis,  etc.;  what  source  documents  describe 
those  terms 

4.  Aircraft  Modeling:  how  the  launch  aircraft  is  physically  described  (presented 
area,  vulnerable  area);  what  maneuvers  are  assumed  before,  during  and  after 
weapon  release;  how  the  launch  aircraft  flight  path  is  determined;  how  the  target 
flight  path  is  determined  (for  air-to-air  weapons) 

5.  Weapon  Modeling:  how  the  weapon’s  trajectory  is  modeled  after  release;  how 
the  warhead  fragments  and  weapon  debris  are  modeled,  and  how  those  data  are 
obtained;  how  the  safety  and  arming  device  is  modeled;  and  whether  debris  from 
the  target  is  included  in  the  analysis 

6.  M&S  and  Credibility:  safe-separation/safe-escape/safe-arming  M&S  used  in 
conducting  these  analyses;  what  (if  any)  significant  differences  in  capability  exist 
between  the  various  codes;  what  software  accuracy  (verification)  documentation 
is  available;  what  data  accuracy  documentation  is  available;  what  output  accuracy 
(validation)  documentation  is  available;  whether  the  code  has  been  accredited  by 
any  users;  and  what  usability  support  (user  groups  or  documentation)  is  available 

In  addition  to  these  comparison  criteria,  the  JSF  JSEAS  effort  has  resulted  in  a  document 
that  describes  23  criteria  for  a  common  safety  analysis  methodology  for  all  air-to-ground 
weapons  authorized  for  release  by  JSF.  Those  criteria  were  developed  jointly  between 
NAWCAD,  the  Air  Force  Seek  Eagle  Office,  and  the  United  Kingdom’s  Ministry  of 
Defense  (MOD).  The  JSEAS  document  compares  some  current  approaches  (Seek  Eagle, 
NAWCAD,  UK)  for  those  23  criteria,  and  documents  the  proposed  JSF  approach  in  each 
case.  Since  the  JSEAS  study  as  reported  only  applies  to  air-to-ground  weapons 
(primarily  gravity  weapons),  the  criteria  have  a  slightly  different  focus  than  originally 
envisioned  for  the  study  reported  here,  which  includes  air-to-air  weapons  as  well.  Also, 
the  JSEAS  criteria  are  not  focused  on  M&S  issues,  but  rather  on  joint  safety  criteria.  As 
a  result,  the  JSEAS  criteria  are  more  comprehensive  than  the  questions  in  Appendix  A 
when  describing  air-to-ground  weapon  safety  criteria,  but  they  do  not  mention  M&S 
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criteria,  or  address  air-to-air  or  powered  weapons  specifically.  They  also  do  not  address 
assumptions  about  target  maneuvers,  or  risks  from  enemy  weapons. 

Table  1  shows  the  results  obtained  from  interviews  with  NAWCWD  and  AMRDEC 
personnel.  NAWCWD  personnel  have  worked  closely  with  the  Seek  Eagle  Office  for 
powered  weapon  analyses,  so  the  entries  in  the  Seek  Eagle  column  are  based  on  the 
NAWCWD  interviews,  with  some  minor  modification  based  on  briefings  from  an  April 
2006  Seek  Eagle  conference  at  Eglin  AFB.  Consequently,  the  entries  in  the  table  may 
not  reflect  what  Seek  Eagle  personnel  responses  would  have  been.  The  entries  for 
AMRDEC  are  based  on  an  Army  briefing  given  at  the  Seek  Eagle  conference  and  another 
given  earlier  to  the  FESWG,  as  well  as  interviews  with  AMRDEC  analysts.  In  addition, 
comments  were  received  from  the  Aviation  Engineering  Directorate  of  AMRDEC  and 
incorporated  into  this  report. 


Assumptions 


The  assumptions  made  about  aircraft  maneuvers,  variations  in  weapon  characteristics, 
safety  and  arming  device  function,  and  any  parameters  that  are  unknown  quantities  can 
often  be  significant  drivers  of  the  results  of  this  type  of  analysis.  We  were  unable  to 
obtain  much  information  about  the  general  assumptions  made  by  NAWCAD,  but  it 
appears  that  the  Seek  Eagle  Office  and  the  NAWCWD  analysts  generally  make  similar 
assumptions  in  these  areas,  and  have  come  to  some  agreement  on  general  approaches  for 
powered  air  weapons  systems.  They  generally  assume  that  straight  and  level  flight  of  the 
launch  aircraft  is  the  “worst  case”  scenario  from  a  safety  standpoint,  since  in  that  case  the 
aircraft  is  following  behind  the  weapon.  That  assumption  may  not  hold,  of  course,  for 
air-to-air  weapons  where  the  target  is  maneuvering,  or  for  off-boresight  launch.  For  air- 
to-air  weapons,  varying  after-launch  fixed-g  maneuvers  are  evaluated.  Launch  altitudes 
and  speeds  are  chosen  from  tactics  manuals,  which  is  likely  to  be  the  case  with  analyses 
conducted  by  all  the  Service  agencies.  Variations  in  weapon  trajectory  are  handled  by 
varying  launch  modes,  weapon  angle  of  attack  and  motor  temperature  (where  data  are 
available).  And  the  assumed  arming  time/distance  takes  into  account  the  tolerances  on 
fuze  design. 

The  Army  approach  seems  to  be  focused  on  specific  low-altitude  launch  tactics  for 
helicopters  against  ground  targets:  hover,  bank,  dive,  break  turns  toward  masking  terrain 
after  launch,  or  vertical  and/or  horizontal  unmask,  then  re -mask  and  egress  after  launch. 
For  the  Army,  the  minimum  safe  range  is  a  combination  of  altitude  and  down  range  from 
the  helicopter  to  its  ground  target.  Army’s  helicopters  normally  fire  weapons  from  30ft 
to  150ft  altitude  and  airspeed  between  hover  and  90  knots.  For  running  cases  (level  flight 
or  diving),  the  common  practice  after  releasing  the  weapon  is  either  veer  to  the  left  or 
veer  to  the  right  and  get  away  from  target. 
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Requirements 


There  are  some  known  differences  between  the  requirements  used  by  the  various  service 
agencies  that  conduct  these  analyses.  One  significant  difference  between  the  Seek  Eagle 
Office  and  NAWCWD  is  that  the  Navy  analysts  consider  probability  of  kill  (Pkm)  of  the 
launch  aircraft  as  a  fallback  metric  to  probability  of  hit  (Phit).  It  appears  that  the  UK  also 
uses  a  similar  metric  to  Pkm  in  their  “self-damage”  probability,  and  the  Army  allows  for 
calculation  of  Pkm  as  part  of  their  risk  assessment  process.  While  Pkm  (or  self  damage)  as 
a  metric  is  less  conservative  from  a  safety  standpoint  than  Phit,  it  follows  the  guidelines  of 
the  original  Joint  Fuze  Management  Board  Agreement  on  safe  separation  analysis 
requirements1.  This  approach  is  best  described  by  a  graphic  from  a  NAWCWD 
viewgraph  presentation2,  shown  in  Figure  1. 

Figure  1.  Safe  Escape  Analysis  Requirements 


Safe  Escape  Analysis  Requirements 


Fuze  Management  Board  Joint  Agreement  on  Safe  Separation  Analysis  for 
Air-Launched  Munitions  (Signed  by  the  Army,  Navy  and  Air  Force  on  23 
February  1978) 


1  Fuze  Management  Board  Joint  Agreement  on  Safe  Separation  Analysis  for  Air  Launched  Munitions ,  23 
Feb  1978 

2  Safe  Escape  Analysis  Overview ,  Naval  Air  Systems  Command  (NAVAIR)  Warfare  Analysis  Department, 
Systems  Division,  AIR  4.10.2,  18  August  2003 
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The  process  is  a  series  of  questions:  if  the  probability  of  hit  is  below  the  one-in- ten- 
thousand  threshold,  then  the  system  meets  the  basic  requirement.  If  that  is  not  the  case,  if 
the  probability  of  hit  (given  detonation  at  or  after  arming)  multiplied  by  the  probability  of 
detonation  at  arming  is  less  than  .0001,  then  it  meets  the  threshold.  In  Figure  1,  Pdet  is 
defined  as  the  probability  of  detonation  at  arming  and  cannot  be  less  than  .01  for  this 
calculation. 

If  neither  of  those  conditions  is  met,  then  probability  of  kill  is  substituted  for  Phit.  The 
following  language  in  the  Joint  Fuze  Management  Board  Agreement  justifies  this 
approach: 

“If  the  minimum  safe-separation  distance  resulting  from  the  above  procedure  restricts 
tactical  delivery  conditions,  the  probability  of  a  fragment  hit  may  be  further  qualified 
by  considering  only  the  presented  area  of  critical  systems  or  components  rather  than 
the  area  of  the  complete  launching  system.”3 

The  NAWCWD  approach  interprets  “probability  of  hit  on  presented  area  of  critical 
systems  or  components”  to  be  represented  by  Pkin.  If  the  system  still  does  not  meet  the 
criterion  using  Pkm,  then  the  Joint  Agreement  allows  for  an  analysis  of  other  risks  and 
hazards;  for  example,  with  an  air-to-air  weapon,  if  the  arm  time  is  too  long,  there  is  some 
risk  that  the  enemy  aircraft  may  be  able  to  launch  a  weapon  inside  our  weapon’s 
minimum  range.  Thus  the  hazard  from  enemy  weapons  may  exceed  the  hazard  from  our 
own  weapon,  and  justify  a  shorter  arm  time.  The  actual  wording  in  the  Joint  Agreement 
on  this  subject  is  as  follows: 

“If  the  above  procedures  still  result  in  restricting  tactical  delivery  conditions,  then 
selected  fuze  arming  conditions  which  are  such  that  a  safe- separation  distance  is  not 
achieved  must  be  justified  by  a  thorough  analysis.  This  analysis  should  consider 
probability  of  a  specific  type  of  damage,  decreased  risk  from  enemy  ordnance,  and 
tactical  advantage  gained  by  use  of  the  recommended  fuze  arming 
characteristics... The  results  of  this  analysis  will  be  included  in  the  final  safe- 
separation  analysis  report  and  the  tactical  manuals  will  identify  those  fuze  arming 
conditions  which,  for  given  delivery  conditions  result  in  specified  hazards  to  the 
launching  system.”4 

There  is  a  natural  complementary  relationship  between  the  system  safety,  reliability,  and 
survivability  disciplines,  illustrated  in  Figure  2.  The  reliability  discipline  evaluates  the 
likelihood  and  the  effects  of  natural  operating  failures,  environmental  factors  and 
operator-induced  events.  The  survivability  discipline  assesses  man-made  hostile  events 
and  their  influence  on  the  system’s  ability  to  perform  its  mission.  And  the  system  safety 
discipline  assesses  the  effects  of  both  hostile  and  normal  environmental  events  on  the 
system.  In  particular,  the  survivability  discipline  evaluates  the  vulnerability  of  the 
launch/release  aircraft  to  damage  by  weapon  debris  fragments.  Safe  escape  analyses  can 


3  Fuze  Management  Board  Joint  Agreement  on  Safe  Separation  Analysis  for  Air  Launched  Munitions,  23 
Feb  1978,  Page  3,  paragraph  4b 

4  Ibid,  Page  4,  paragraph  4d 
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make  use  of  the  vulnerability  assessments  that  are  a  natural  byproduct  of  survivability 
analyses,  and  in  fact  that  is  what  NAWCWD  and  AMRDEC  analysts  use  as  input  to  their 
probability  of  kill  assessments.  As  a  result,  the  Pkiii  assessments  that  support  safe  escape 
analyses  are  based  on  high-fidelity  models  of  the  launch  aircraft. 


Figure  2.  Relationships  Between  Safety,  Survivability  and  Reliability 


The  Seek  Eagle  office  does  not  follow  this  approach;  their  analysts  use  only  Phit  as  a 
metric.  From  anecdotal  evidence  it  appears  that  the  same  is  true  of  NAWCAD  analysts. 

It  should  be  noted  that  a  “fragment  hit”  is  defined  in  the  Joint  Agreement  as: 

“A  fragment  which  contains  sufficient  kinetic  energy  to  penetrate  the  launch  aircraft 
skin  which  is  exposed  to  the  hazard.  Caution  must  be  exercised  not  to  eliminate  from 
the  calculations  those  low  relative  velocity  fragments  which  may  cause  serious 
damage  if  ingested  by  the  engine(s).”5 

Other  than  the  Army,  it  is  not  clear  whether  the  service  agencies  that  conduct  these 
analyses  restrict  fragment  sizes  and  velocities  to  those  conditions  when  calculating 
probability  of  hit.  Anecdotal  evidence  seems  to  indicate  that  very  small  fragment  sizes 
are  included  in  most  of  these  analyses,  which  could  result  in  overly  conservative  risk 
assessments.  The  Army,  on  the  other  hand,  restricts  the  hit  analysis  to  only  include 
fragments  with  kinetic  energy  of  5  ft-lbs  or  more  relative  to  the  aircraft,  or  for  which  the 
striking  velocity  is  above  V50  (velocity  at  which  half  of  the  fragments  penetrate  the 
aircraft  skin)  -  the  two  different  criteria  are  used  by  the  two  different  directorates  at 
AMRDEC  that  conduct  these  analyses. 

Based  on  the  results  of  safe  escape/safe  arming  analyses  conducted  by  NAWCWD,  there 
have  occasionally  been  post-launch  maneuver  requirements  placed  on  the  launch  aircraft 
in  order  to  meet  the  safety  criterion.  This  is  only  true  for  air-to-ground  weapons,  as 


5  Ibid,  Page  1,  paragraph  2c 
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opposed  to  air-to-air  weapons.  There  is  some  evidence  that  this  is  also  the  case  for  Seek 
Eagle,  UK  and  NAWCAD  analysis  results.  It  is  unknown  whether  the  Army  places 
similar  restrictions  on  their  weapons  delivery  helicopters.  A  briefing  to  the  FESWG  by 
AMRDEC  analysts  from  the  System  Simulation  and  Development  Directorate  seems  to 
indicate  that  they  do  not  place  such  restrictions6.  That  briefing  also  confirms  that  the 
AMRDEC  analysts  from  that  directorate  use  the  one  in  ten  thousand  probability 
requirement. 

The  requirement  for  probability  of  hit  (or  kill)  being  less  than  one  in  ten  thousand  seems 
to  have  been  based  on  historical  loss  rates  and/or  mishap  rates  from  the  era  of  the 
Vietnam  conflict.  The  initial  analysis  of  that  data  appears  to  have  been  reported  in  a 
letter  dated  in  1973,  which  led  to  the  eventual  signing  of  the  Joint  Fuze  Management 
Board  Agreement  in  1978.  We  were  unable  to  retrieve  a  copy  of  the  1973  letter; 
however,  a  more  recent  analysis  of  mishap  rates  for  both  tactical  aircraft  and  unmanned 
aircraft  was  reported  in  the  UAS  Roadmap  20057.  Figure  3,  reproduced  from  that 
document,  shows  mishap  rates  for  F-16,  Global  Hawk,  Predator  and  other  unmanned 
aircraft  systems  (UAS)  as  a  function  of  cumulative  flight  hours.  For  the  F-16  and  the 
Predator,  which  both  have  over  100,000  flight  hours  logged,  the  mishap  rate  appears  to 
be  on  the  order  of  ten  mishaps  per  100,000  flight  hours;  most  of  the  other  systems  seem 
to  be  converging  on  a  similar  mishap  rate  as  their  cumulative  flight  hours  increase. 


It  is  difficult  to  reconstruct  data  on  probability  of  hit  by  threat  action  for  the  Vietnam  era 
(or  for  more  recent  conflicts,  for  that  matter).  One  analysis  done  by  the  Air  Force  using 
data  from  the  Survivability/Vulnerability  Information  Analysis  Center  (SURVIAC) 
shows  that  USAF  aircraft  hit  rates  per  sortie  in  Vietnam  ranged  from  one  in  100  (for  F-4 


6  Safe  Separation  Analysis,  Kim  Williams,  Shane  Strickland,  Brent  Deerman,  30  Nov  2005 

7  Unmanned  Aircraft  Systems  Roadmap  2005-2030,  Office  of  the  Secretary  of  Defense,  4  Aug  2005 
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and  AC-130  aircraft)  to  one  in  4,000  (for  the  C-130):  see  Figure  4.  The  wide  range  in 
probability  of  hit  is  explained  by  the  fact  that  the  relative  exposure  of  the  aircraft  to  threat 
systems  varied  considerably  by  mission  in  that  conflict.  Gunships  and  fighters  were  far 
more  likely  to  be  directed  to  areas  where  they  encountered  threat  systems  than  were 
transport  aircraft.  For  none  of  the  aircraft  of  that  era  was  the  probability  of  hit  per  sortie 
less  than  two  in  10,000:  they  tended  more  toward  one  in  100  or  one  in  1,0008. 


Figure  4.  Sorties  Per  Hit  in  South-East  Asia 


We  obtained  a  report  by  the  Center  for  Naval  Analyses  that  reported  Vietnam  combat 
damage  incidents  and  aircraft  kill  rates  for  USN  and  USMC  fixed  wing  aircraft9.  In  that 
report,  the  damage  incident  and  kill  rates  for  USN  aircraft  were  2,147  damage  incidents 
and  538  loss  incidents  per  512,757  sorties,  or  a  hit  rate  of  5.23  per  1000  sorties  and  a  kill 
rate  of  1.05  per  1000  sorties.  For  USMC  aircraft,  there  were  1,871  damage  incidents  and 
173  kills  in  323,542  sorties,  or  hit  rate  of  6.32  per  1000  sorties  and  a  kill  rate  of  0.54  per 
1000  sorties  (note  that  damage  incidents  plus  kills  add  up  to  hit  incidents).  These  data 
were  for  the  period  from  April  1965  through  March  1973.  Thus,  the  overall  average  hit 
rates  were  on  the  order  of  one  hit  per  100  sorties,  and  the  kill  rates  on  the  order  of  one 
loss  per  1000  sorties. 

Another  analysis  by  SURVIAC  of  aircraft  hits  during  Desert  Storm10  (see  Figure  5) 
indicates  that  hit  rates  in  that  conflict  also  varied  by  mission  type:  for  interdiction 
missions,  there  were  about  30  hits  in  33,000  sorties  (about  one  in  1,000);  for  close-air- 
support  (CAS)  missions,  there  were  12  hits  in  fewer  than  5,000  sorties  (about  one  in  400). 
Suppression  of  Enemy  Air  Defense  (SEAD)  missions  resulted  in  5  hits  for  approximately 
5,000  sorties  (one  in  1,000),  whereas  Defensive  Counter  Air  (DCA)  missions  resulted  in 


8  Historical  Combat  Data,  briefing  given  by  Kevin  Crosthwaite,  Director,  Survivability-Vulnerability 
Information  Analysis  Center  (SURVIAC),  Aircraft  Survivability  Short  Course,  11-13  July  2006 

9  U.S.  Navy,  Marine  Corps  and  Air  Force  Fixed  Wing  Aircraft  Losses  and  Damage  in  Southeast  Asia 
(1962-1973),  Michael  M.  McCrea,  Center  for  Naval  Analyses,  CRC  305,  August  1976 

10  Mission  &  Campaign  Analysis,  briefing  given  by  Kevin  Crosthwaite,  Director,  Survivability- 
Vulnerability  Information  Analysis  Center  (SURVIAC),  Aircraft  Survivability  Short  Course,  11-13  July 
2006 
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no  hits  for  7,500  sorties  (because  there  were  virtually  no  air-to-air  threats  that  would  fly 
in  that  conflict).  This  analysis  seems  to  imply  that  the  hit  rate  due  to  hostile  action 
during  Desert  Storm  was  on  the  order  of  one  hit  in  1,000  sorties. 


Figure  5.  Hits  During  Desert  Storm  by  Mission  Type 


Phit  per  sortie  ~  1 0'3 

(or  zero,  for  DCA  -  no  threat  a/c) 


Source:  SURVIAC 


Comparing  either  mishap  rates  (per  flight  hour)  or  probability  of  hit  during  historical 
conflicts  (per  sortie)  to  probability  of  being  hit  by  your  own  weapon  given  launch  is 
really  comparing  apples  and  oranges.  We  have  mishaps  per  flight  hour;  threat  hits  per 
sortie;  and  in  the  safe  escape  case,  fragment  hits  per  weapon  launch.  However,  in  a 
system  safety  sense,  both  mishap  rate  and  combat  hit  rate  provide  some  measure  of  how 
“safe”  each  of  these  activities  are:  in  the  one  case,  what  are  historical  “acceptable” 
mishaps  during  flight  operations,  and  in  the  other  what  are  the  historical  hit  rates  that 
have  been  tolerated  each  time  an  aircraft  is  sent  on  a  mission.  As  such,  they  provide  at 
least  a  qualitative  measure  of  accepted  safety  levels. 

Using  probability  of  hit  less  than  0.0001  as  a  safety  criterion  seems  to  be  a  reasonable  fit 
with  historical  mishap  rate  data  and  with  more  recent  data  as  reported  for  both  manned 
and  unmanned  systems.  It  is  less  consistent  with  combat  hit  rates  per  sortie,  which  are  on 
the  average  at  least  an  order  of  magnitude  higher  (one  in  1,000  for  Desert  Storm,  and  for 
USN  and  USMC  fixed  wing  aircraft  in  Vietnam,  closer  to  one  in  100).  Those  higher 
combat  hit  rates  do,  however,  provide  justification  for  considering  an  analysis  of 
“...probability  of  a  specific  type  of  damage,  decreased  risk  from  enemy  ordnance,  and 
tactical  advantage  gained  by  use  of  the  recommended  fuze  arming  characteristics...”  in 
the  system  safety  assessment,  as  allowed  by  the  Joint  Fuze  Management  Board 
Agreement.  If  the  hit  (or  kill)  probability  does  not  meet  the  one  in  10,000  requirement, 
then  an  analysis  of  other  risks  to  the  system  should  be  conducted  to  determine  if  they  out- 
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weigh  the  risk  of  damage  from  launching  our  own  weapon.  Historically,  those  hit  rates 
from  threat  weapons  have  been  much  higher. 

Definitions 


Much  confusion  has  resulted  from  multiple  definitions  of  terms  in  this  area  over  the  last 
30  years  and  more.  “Safe  Separation”  in  particular  has  been  used  for  more  than  one 
purpose.  “Safe  separation”  has  been  taken  to  mean  both  safe  release  of  the  weapon  from 
its  launcher,  and  that  the  weapon  is  a  safe  distance  away  from  the  launcher  at  the  time  the 
fuze  arms  the  warhead.  Those  are  two  completely  different  concepts,  but  have  both  been 
called  the  same  thing  in  the  past.  Official  documentation  doesn’t  seem  to  help  the 
problem,  as  can  be  seen  from  the  published  definitions  below: 

Definition  from  MIL-HDBK-176311: 

Safe  Separation:  The  parting  of  a  store(s)  from  an  aircraft  without  exceeding  the 
design  limits  of  the  store  or  the  aircraft  or  anything  carried  thereon,  and  without 
damage  to,  contact  with,  or  unacceptable  adverse  effects  on  the  aircraft,  suspension 
equipment,  or  other  store(s)  both  released  and  unreleased. 

Definition  from  MIL-STD-1316E12: 

Safe  Separation  Distance:  The  minimum  distance  between  the  delivery  system  (or 
launcher)  and  the  launched  munition  beyond  which  the  hazards  to  the  delivery  system 
and  its  personnel  resulting  from  the  functioning  of  the  munition  are  acceptable. 

Definition  from  the  Joint  Agreement13: 

Safe-Separation  Distance:  the  minimum  distance  between  the  launching  system 
(AIRCRAFT  &  PILOT)  and  its  launched  munitions  at  which  hazards  associated  with 
munitions  functioning  are  acceptable.  This  distance  may  be  achieved  by  providing 
arming  delays(s)  (time  or  distance). 

It  could  be  argued  that  the  MIL-HDBK  definition  of  “safe  separation”  is  distinct  from  the 
two  (similar)  definitions  of  “safe  separation  distance”.  However,  the  use  of  the  words 
“safe  separation”  in  all  three  definitions  is  the  cause  of  the  confusion,  especially  when 
assessments  of  the  two  have  been  called  “safe  separation  analysis”  in  the  past. 

With  regard  to  definitions,  all  of  the  service  experts  now  seem  to  use  “Safe  Escape 
Analysis”  to  describe  the  work  they  do  in  arriving  at  risk  assessments  for  air-launched 


11  Aircraft/Stores  Compatibility:  Systems  Engineering  Data  Requirements  And  Test  Procedures,  MIL- 
HDBK-1763 

12  Department  of  Defense  Design  Criteria  Standard,  Fuze  Design,  Safety  Criteria  Fox,  10  July  1998,  MIL- 
STD-1316E 

13  Fuze  Management  Board  Joint  Agreement  on  Safe  Separation  Analysis  for  Air  Launched  Munitions,  23 
Feb  1978,  Page  1,  paragraph  2a 
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weapon  systems.  The  details  of  that  analysis  differ,  however.  Seek  Eagle  and 
NAWCAD  analysts  appear  to  define  safe  escape  analysis  as  producing  a  Minimum 
Release  Altitude  (MRA)  consistent  with  a  maximum  probability  of  hit  threshold;  MRA  is 
termed  Minimum  Safe  Release  Altitude  for  Fragment  Avoidance,  or  MinAlt  by 
NAWCWD.  NAWCWD,  on  the  other  hand,  uses  the  term  “safe  escape  analysis”  to  refer 
to  an  assessment  of  the  risk  of  launching  an  air  weapon,  and  does  not  refer  to  a  minimum 
safe  release  altitude,  probably  since  NAWCWD  only  analyzes  powered  weapons.  The 
AMRDEC  analysts  define  it  as  “determining  the  minimum  safe  range  for  helicopters  to 
release  weapons.”  They  also  feel  that  any  definition  of  “Safe  Escape”  should  include 
both  altitude  and  down  range,  since  to  them  the  minimum  safe  range  is  a  combination  of 
altitude  and  down  range  from  the  helicopter  to  its  ground  target. 

MIL-HDBK-1763  provides  a  definition  for  safe  escape  that  may  be  part  of  the  solution, 
but  that  still  concatenates  the  two  terms  “safe  escape”  and  “safe  separation”: 

Safe  Escape/Safe  Arming:  Safe  escape  is  the  minimum  release  altitude  that  will 
provide  the  delivery  aircraft  acceptable  protection  from  weapon  fragmentation  for 
detonation  at  the  preplanned  point.  Safe  arming  separation  is  the  selection  of  a 
minimum  safe  fuze  arm  time  setting  that  will  provide  the  delivery  aircraft  acceptable 
protection  from  weapon  fragmentation  if  early  detonation  should  occur. 


Aircraft  Modeling 


Probability  of  hit  on  the  aircraft  by  weapon  debris  fragments  is  calculated  using  a  simple 
six-view  “box”  representation  of  the  presented  area  of  the  aircraft.  This  is  true  of  all  the 
approaches  used  by  the  services,  except  that  the  AMRDEC  process  provides  for  a  second 
pass  with  a  detailed  CAD  model  of  the  aircraft:  their  first  pass  with  the  “shoe-box  model” 
is  intended  to  compute  “potentially-hit-fragments”.  Since  their  approach  is  a  set  of  one 
million  Monte-Carlo  iterations  of  the  warhead  detonation,  the  initial  screening  pass  is 
needed  to  reduce  the  computations  required  for  the  detailed  CAD  model  of  the  aircraft  to 
only  those  fragments  with  some  probability  of  actually  hitting  it.  The  second  pass 
actually  calculates  the  probability  of  hit  on  the  aircraft. 

For  the  NAWCWD  approach  to  determining  probability  of  kill,  the  presented  areas  are 
replaced  by  six-view  vulnerable  areas  obtained  from  vulnerability  analyses  conducted  by 
the  NAWCWD  Survivability  Division  for  the  launch  aircraft  in  question.  Vulnerable 
area  is  defined  as  probability  of  kill  given  a  hit  multiplied  by  presented  area; 
consequently,  it  is  an  “expected  value”  approach  to  describing  system  vulnerability.  In  a 
way,  it  represents  the  presented  area  of  critical  components  and  systems.  Vulnerable  area 
is  a  function  of  fragment  size  and  striking  velocity. 

In  the  case  of  air-to-air  weapons,  NAWCWD  analysts  assume  that  straight  and  level 
flight  for  the  target  aircraft  is  the  worst  case,  since  in  that  case  the  launch  aircraft  is  most 
likely  to  follow  directly  behind  the  weapon.  NAWCWD  and  Seek  Eagle  analysts  use 
the  JTCG/ME  Joint  Air-to-Air  Model  (JAAM)  to  develop  flight  paths  for  both  launch 
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aircraft  and  target  aircraft  (for  air-to-air  weapons).  In  addition,  for  the  Air  Force  the 
Aircraft/Weapon  Delivery  Software  (AWDS)  library  can  be  used  with  CASES  to  develop 
aircraft  flight  paths.  No  information  was  available  on  NAWCAD  aircraft  trajectory 
modeling.  For  AMRDEC  analyses,  flight  data  (aircraft  velocity  components  and 
orientation)  are  generated  with  RCAS  or  FlightLab,  both  of  which  are  high  fidelity 
helicopter  simulation  software.. 


Weapon  Modeling 


For  the  NAWCWD  and  Seek  Eagle  analyses,  the  trajectory  of  the  weapon  is  almost 
always  provided  by  the  weapon  systems  program  office,  either  directly  or  via  delivery  of 
their  weapon  flight  simulation;  for  powered  weapons  the  simulation  is  almost  always  a 
six  degree-of-freedom  simulation,  providing  position,  yaw,  pitch  and  roll  as  a  function  of 
time  after  launch.  The  Seek  Eagle  approach  also  allows  for  weapon  trajectories  to  be 
developed  by  the  AWDS  dynamically  linked  library  as  part  of  CASES.  For  the  Army, 
weapon  trajectories  are  provided  by  program  office  6-dof  simulations  and  input  into  the 
ADAMS  software. 

Generally  speaking,  the  weapon  trajectory  simulations  provided  by  the  program  office 
responsible  for  the  weapon’s  development  are  considered  to  be  “ground  truth”  and  are 
usually  rigorously  validated,  even  if  those  validation  data  are  not  always  documented  or 
retrievable. 

Weapon  system  debris  fragments  and  warhead  fragments  are  modeled  using  polar  (and  in 
some  cases  azimuth)  zones,  usually  with  constant  fragment  ejection  velocity  over  the 
zone  and  an  average  number  of  fragments  per  zone.  The  size  of  the  fragment  zones 
depends  on  the  quality  of  the  arena  test  data  for  the  specific  weapon  in  question; 
generally  10-degree  polar  zones  are  used,  and  occasionally  the  data  will  allow  for  5- 
degree  polar  zones.  CASES  allows  a  maximum  of  18  polar  zones  and  exactly  24  roll 
zones  (of  15  degrees).  For  most  systems,  the  fragment  zones  are  symmetric  around  the 
body  of  the  weapon.  It  is  possible  to  conduct  separate  analyses  for  fragments  of  unusual 
size  and/or  velocity  (warhead  fragments,  for  example,  or  bomb  lugs),  and  combine  the 
results  of  that  analysis  with  the  main  body  of  fragments  into  an  aggregate  result. 
Gamma  zones  are  the  mechanism  that  accounts  for  fragment  size  and  shape,  where 
gamma  is  the  ratio  of  a  fragment’s  average  presented  area  to  its  mass. 

Without  discussing  this  in  person  with  the  individual  service  experts,  we  were  not  able  to 
determine  with  what  fidelity  all  service  weapons  debris  patterns  are  modeled.  However, 
we  did  obtain  a  briefing  and  paper  that  were  presented  at  an  April  2006  Seek  Eagle 
conference  that  describes  the  approach  used  for  Seek  Eagle  fragmentation  test  data 
collection  and  analysis14.  That  paper  described  the  use  of  a  computer-driven  Fragment 
Digitizer  System  that  greatly  facilitates  development  of  fragment  shape  factors,  presented 


14  Common  Advanced  Safe  Escape  System  (CASES):  A  Look  Behind  the  Scene,  Tommy  Collins,  James 
Burton,  Shane  Sartalamacchia,  Tama  Leach 
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areas,  and  gamma  values.  The  weapon  debris  model  is  a  principal  driver  of  the  results  of 
the  safe  escape/safe  arming  analysis. 


The  Army  briefing  from  the  Seek  Eagle  conference15  shows  example  polar  zones  for  a 
2.75  inch  rocket  (M151  Warhead  and  rocket  debris);  the  data  in  the  briefing  show  5- 
degree  polar  zones,  and  those  are  assumed  to  be  representative  of  the  fidelity  of  Army 
system  debris  models. 


M&S  and  Credibility 


NAWCAD,  NAWCWD  and  Seek  Eagle  all  use  safe  escape  models  and  simulations  that 
have  common  heritage.  NAWCAD  uses  Path  4,  NAWCWD  uses  the  Advanced  Safe 
Escape  Program  (ASEP),  and  Seek  Eagle  uses  the  Common  Advanced  Safe  Escape 
System  (CASES),  all  of  which  have  their  origins  in  the  development  of  Path  2  by  the 
Navy  at  Dahlgren,  VA.  Path  4  is  an  evolution  from  Path  2,  Path  3D  was  an  evolution 
from  Path  2,  ASEP  is  an  evolution  from  Path  3D,  and  CASES  is  an  evolution  from 
ASEP. 

All  of  those  methodologies  use  similar  (if  not  the  same)  representations  of  missile  debris 
and  warhead  fragments,  fragment  flight  paths,  calculations  of  probability  of  hit  (and  kill), 
and  launch  aircraft  representations.  There  are  some  capabilities  that  have  been  added 
with  each  evolution  of  the  code:  for  example,  CASES  appears  to  be  ASEP  with  a  better 
Graphical  User  Interface  (GUI)  and  some  pre-generated  warhead  data  files  for  certain 
weapons.  It  also  allows  for  calculation  of  “deconfliction”,  meaning  that  it  will  determine 
whether  fragments  can  hit  another  aircraft  in  the  same  flight  as  the  launch  aircraft.  ASEP 
added  asymmetric  fragment  roll  zones  to  PATH-3D  along  with  some  other 
improvements. 

There  has  been  no  organized  effort  to  conduct  and  document  verification  and  validation 
on  any  of  the  codes.  There  have  been  comparison  runs  made  between  CASES  and 
ASEP,  with  some  minor  errors  corrected  as  a  result  of  those  runs;  however,  there  is  no 
documentation  available  of  those  comparisons  or  the  changes  that  were  made  as  a  result. 
There  is  anecdotal  evidence  that  the  Seek  Eagle  office  has  accredited  both  CASES  and 
ASEP  for  individual  weapons  systems  analyses,  but  no  documentation  of  those 
accreditation  decisions  was  available. 

The  Seek  Eagle  office  provides  limited  support  to  users  of  both  ASEP  and  CASES.  A 

i  /:  i  n 

User  Manual  and  Analyst  Manual  are  available  for  ASEP.  It  is  unknown  what 
documentation  is  available  for  CASES  or  for  Path  4. 


15  Safe  Escape  for  the  Army  Helicopters ,  Tuan  Pham,  Aviation  &  Missile  Research,  Development  & 
Engineering  Center,  Aviation  Engineering  Directorate,  Weapons  &  Sensors  Branch,  April  2006 

16  Advanced  Safe  Escape  Program  (ASEP)  Users  Manual ,  ASEP-UM-002,  December  1996,  Tybrin 
Corporation 

17  Advanced  Safe  Escape  Program  (ASEP)  Analysts  Manual ,  ASEP-AM-002,  December  1996,  Tybrin 
Corporation 
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The  Army  analysts  at  AMRDEC  use  a  set  of  simulations  that  were  developed 
independently  from  the  Seek  Eagle  set  of  models.  They  use  the  Army  Safe  Escape 
Analysis  Tool  (ASEAT),  which  includes  several  additional  software  packages:  ADAMS, 
EASY5,  FlightLab,  3D-CAD  and  weapon  fly-out  models.  ADAMS  is  the  6-dof 
simulation  used  for  aircraft  flight  paths  and  fragment  trajectories.  EASY5  is  a  system- 
level  modeling  tool  used  for  describing  the  physical  components  of  the  aircraft  system, 
such  as  hydraulics,  controls  and  electrical  subsystems.  The  3D-CAD  simulation  provides 
a  detailed  physical  description  of  the  aircraft  that  is  used  in  the  final  Phit  calculation. 
ADAMS,  EASY5,  and  FlightLab  are  commercial  software.  RCAS  and  weapons  6-dof 
are  the  Army’s  software. 

If  there  is  a  need  to  fire  the  munition  at  a  target  at  a  range  closer  than  the  minimum  safe 
range,  then  Pkiii  can  be  calculated  as  a  safety  metric  with  a  simulation  such  as  the 
Advanced  Joint  Effectiveness  Model  (AJEM).  Ptm  is  an  important  piece  of  information 
used  in  conducting  risk  assessments  for  the  Army  Airworthiness  Qualification  and 
Release  processes. 

The  primary  differences  between  the  AMRDEC  and  Seek  Eagle  simulations  is  that  the 
AMRDEC  approach  uses  a  one -million  iteration  Monte  Carlo  simulation  of  weapon 
debris  fragmentation,  and  a  two-pass  calculation  of  probability  of  hit  (to  reduce  run¬ 
time).  The  Seek  Eagle  models,  on  the  other  hand,  use  an  expected  value  approach, 
computing  an  expected  number  of  fragments  in  each  polar-azimuth  zone.  The  Army  Phit 
calculation  also  uses  a  detailed  CAD  model  of  the  aircraft,  whereas  the  other 
methodologies  use  a  simple  six-sided  representation  of  the  aircraft’s  presented  area.  It  is 
not  clear  whether  the  fidelity  of  the  weapon  debris  data  obtained  from  arena  testing  is 
consistent  with  the  detail  available  in  CAD  models  of  the  aircraft.  It  may  be  that  the 
shorter  range  weapons  employed  by  the  Army  result  in  close  enough  detonations  to 
justify  the  need  for  the  additional  fidelity  in  the  aircraft  model. 

There  is  no  documented  verification,  validation  or  accreditation  of  the  ASEAT  set  of 
simulations,  nor  is  there  documentation  available  for  ASEAT. 
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CONCLUSIONS 


Assumptions:  There  was  very  little  information  available  about  assumptions  made  by 
analysts  other  than  those  at  NAWCWD.  Since  these  assumptions  can  drive  the  answers, 
it  is  important  that  they  be  consistent  across  the  service  agencies  that  conduct  these 
analyses. 

Requirements:  The  requirement  for  probability  of  less  than  one  in  ten  thousand  appears 
to  be  consistent  with  historical  mishap  rates  (per  flight  hour),  but  it  is  not  as  consistent 
with  combat  hit  rates  (per  sortie),  which  are  an  order  of  magnitude  higher.  The  principal 
difference  between  the  service  agencies  in  requirements  is  the  use  of  Pkiii  as  a  fallback 
metric  to  Phit.  Analysts  at  NAWCWD  use  probability  of  kill  (and  the  UK  analysts  use 
something  similar),  and  AMRDEC  uses  Pkiii  as  part  of  their  risk  assessment  process,  but  it 
appears  that  Seek  Eagle  and  NAWCAD  strictly  use  Phit.  The  JSEAS  project  has  put 
considerable  effort  into  examining  requirements  across  the  services  and  coming  up  with 
common  requirements  for  the  JSF  program.  That  project  has  not  involved  the  Army, 
however,  since  there  is  no  Army  variant  of  JSF.  Consequently,  the  requirements 
developed  by  JSF  only  apply  (so  far  at  least)  to  the  Navy  (and  by  extension  the  Marine 
Corps)  and  the  Air  Force.  Also,  the  JSEAS  requirements  document  states  that  they  only 
apply  to  air-to-ground  weapons  (with  an  apparent  emphasis  on  gravity  weapons). 

Definitions:  The  use  of  the  term  “safe  separation  analysis”  to  mean  more  than  one  thing 
causes  considerable  confusion,  especially  for  weapons  system  program  offices  who  may 
think  they’ve  met  a  requirement  only  to  find  that  they  only  addressed  another  issue 
entirely.  All  of  the  current  practitioners  of  the  discipline  seem  to  refer  to  their  work  as 
“safe  escape  analysis”,  which  may  offer  a  solution  to  that  issue. 

Aircraft  Modeling:  Aircraft  presented  area  seems  to  be  modeled  in  the  same  way  by  all 
of  the  services,  using  a  six-view  total  presented  area  (the  Army  adds  a  detailed  CAD 
model  of  the  aircraft  for  their  final  Phit  calculation).  NAWCWD  analysts  also  use  six- 
view  vulnerable  area  to  represent  “the  presented  area  of  critical  components  and 
systems”.  AMRDEC  allows  for  using  a  model  like  AJEM  for  a  detailed  Pkiii  calculation. 
Aircraft  flight  paths  for  NAWCWD  and  Seek  Eagle  are  based  on  the  JTCG/ME  JAAM 
methodology;  AMRDEC  uses  RCAS  or  FlightLab.  It  is  unknown  what  methodology 
NAWCAD  uses. 

Weapon  Modeling:  Weapon  trajectories  are  usually  generated  using  program  office  6- 
dof  flight  simulations  for  powered  weapons  systems.  We  were  unable  to  obtain 
information  about  NAWCAD  approaches  to  gravity  weapons  modeling  or  with  what 
fidelity  weapon  fragmentation  is  usually  modeled  at  NAWCAD. 

M&S  and  Credibility:  NAWCWD,  NAWCAD  and  Seek  Eagle  all  use  M&S  which 
evolved  from  the  same  original  source,  Path-2,  which  was  developed  originally  by  the 
Navy  at  Dahlgren,  VA.  Based  on  anecdotal  evidence,  the  differences  between  these 
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codes  seem  to  be  minor  compared  to  the  similarities.  There  is  no  available 
documentation  of  past  verification  and  validation  activities.  We  were  only  able  to  obtain 
documentation  of  the  ASEP  model  used  at  NAWCWD.  Seek  Eagle  has  apparently 
accredited  ASEP  and  CASES  for  some  applications,  but  documentation  of  those 
accreditation  decisions  was  not  available.  The  Army  uses  an  independently  developed 
methodology  called  ASEAT,  which  is  a  Monte  Carlo  simulation,  and  uses  detailed 
representations  of  the  launch  aircraft’s  geometry.  There  is  no  documentation  of  any  past 
VV&A  results  for  ASEAT. 
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RECOMMENDATIONS 


Assumptions:  There  should  be  Joint  Service  guidelines  for  the  assumptions  made  in 
conducting  safe  escape/safe  arming  analyses.  In  particular,  guidance  should  be  provided 
regarding  launch  aircraft  maneuvers,  weapon  variations  (angle  of  attack,  motor 
temperature,  roll  orientation,  etc.),  environmental  factors,  safe-arm  device  variations,  and 
other  factors  that  potentially  drive  the  analysis  results. 

Requirements:  The  JSEAS  document  provides  a  comprehensive  set  of  requirements  for 
air-to-ground  weapons  safety  analyses  that  have  been  accepted  by  the  participants  in  JSF. 
Those  requirements  should  serve  as  the  starting  point  for  expansion  to  include  Army 
requirements  and  air-to-air  weapon  system  requirements,  and  to  ensure  that  powered  air- 
to-ground  weapon  requirements  are  adequately  treated.  Provision  should  be  considered 
in  future  Joint  requirements  for  application  of  the  process  outlined  in  the  original  Joint 
Agreement  between  all  the  services,  particularly  the  inclusion  of  Pkiii  as  a  metric  and  the 
provision  for  additional  analyses  to  support  operational  use  of  weapons  that  do  not  meet 
the  0.0001  probability  requirement.  Historical  combat  hit  rates  offer  justification  for 
those  additional  analyses.  Historical  mishap  rates  seem  to  be  consistent  with  the  basic 
safety  requirement  of  Phit  given  weapon  release  of  less  than  one  in  10,000:  there  does  not 
appear  to  be  a  rationale  for  changing  that  requirement. 

Definitions:  A  possible  solution  to  the  definitions  problem  is  to  divide  the  definition  of 
“Safe  Escape/Safe  Arming”  that  is  offered  in  MIL-HDBK-1763  into  separate  definitions 
for  the  two  terms.  Since  all  of  the  current  participants  in  this  type  of  analysis  use  the 
term  “safe  escape  analysis”  vice  “safe  separation  analysis”,  this  would  seem  to  be  in  line 
with  current  practice.  However,  using  these  definitions,  the  analyses  conducted  for  air- 
to-air  weapons  are  better  characterized  as  “safe  arming  analyses”,  since  they  do  not 
generally  involve  determining  a  minimum  release  altitude.  We  offer  an  alternative  below 
that  should  satisfy  both  air-to-air  and  air-to-ground  weapons  systems.  This  change  also 
would  mean  that  the  definitions  of  “safe  separation  distance”  in  both  MIL-STD-1316E 
and  in  the  original  Joint  Fuze  Management  Board  agreement  should  be  changed  to  “safe 
arming  distance”.  Draft  definitions  are  as  follows: 

Safe  escape:  Safe  escape  is  the  required  release  conditions  and  post-launch 
maneuvers  that  will  provide  the  delivery  aircraft  acceptable  protection  from 
weapon  fragmentation  for  detonation  at  the  preplanned  point,  or  at  or  after 
arming;  this  may  result  in  a  minimum  safe  release  altitude  or  safe  release  altitude 
and  down  range  from  the  target. 

Safe  arming:  Safe  arming  is  the  selection  of  a  minimum  safe  fuze  arm  setting 
that  will  provide  the  delivery  aircraft  acceptable  protection  from  weapon 
fragmentation  if  detonation  should  occur  at  or  after  the  fuze  arm  time/distance. 
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Separation:  The  termination  of  all  physical  contact  between  a  store,  or  portions 
thereof,  and  an  aircraft;  or  between  a  store,  or  portions  thereof,  and  suspension 
equipment. 

Safe  separation:  The  parting  of  a  store(s)  from  an  aircraft  without  exceeding  the 
design  limits  of  the  store  or  the  aircraft  or  anything  carried  thereon,  and  without 
damage  to,  contact  with,  or  unacceptable  adverse  effects  on  the  aircraft, 
suspension  equipment,  or  other  store(s)  both  released  and  unreleased. 

Aircraft  Modeling:  There  should  be  agreed-upon  guidelines  for  launch  aircraft  post¬ 
launch  maneuvers  to  consider  for  safety  reasons.  Sensitivity  analyses  should  be 
conducted  to  determine  whether  there  is  a  need  for  more  detailed  aircraft  representations 
than  6-view  presented  areas  (as  in  the  AMRDEC  approach). 

Weapon  Modeling:  There  should  be  agreed-upon  guidelines  for  the  fidelity  of  weapon 
debris  modeling  (polar  zones,  etc.).  Guidelines  should  be  established  for  when  to 
segregate  “unusual”  fragments  for  separate  analysis  (such  as  bomb  lugs,  warhead 
fragments  that  are  likely  to  have  much  higher  velocities  than  debris  fragments,  etc.). 
Fragments  should  only  be  included  in  the  weapon  debris  model  if  they  are  capable  of 
penetrating  the  skin  of  the  aircraft  (per  the  Joint  Agreement  definition  of  “fragment  hit”, 
and  consistent  with  the  Army’s  KE>5  ft- lbs  or  striking  velocity>V5o  requirement  for 
fragment  inclusion  in  the  debris  model). 

M&S  and  Credibility:  Navy  representatives  should  consider  migrating  to  the  latest 
version  of  the  Seek  Eagle  methodology  (CASES).  When  available,  the  JSEAS 
methodology  should  be  assessed  for  adoption  as  the  standard  Joint  Service  methodology. 
Documented  verification  and  validation  evidence  should  be  developed  for  any  M&S  tools 
used  in  safe  escape/safe  arming  analyses.  Documentation  of  all  methodologies  used  by 
the  services  should  be  developed,  maintained  and  distributed  to  users.  An  Accreditation 
Support  Package  (ASP)  should  be  developed  for  any  M&S  tools  that  are  continuing  in 
use. 
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Table  1.  Interview  Results 


Category 

Issue 

NAWCWD 

NAWCAD 

Seek  Eagle 

AMR  me 

Requirements 

Launcher 

Vulnerability 

Metric 

Hit  &  Kill 

Hit  Only 

Hit  Only 

Hit  (KE>5  ft-lbs  or 
V>V50)  &  Kill 

Probability 

Requirement 

0.0001  or  0.01 

Or  outside  hazards 
analysis 

0.0001 

0.0001  or  0.01 

Or  outside  hazards 
analysis 

0.0001 

Maneuver 
after  launch 
required  if 
Prob.  not 
met? 

Yes  (in  one  or  two 
cases) 

No 

Analysis 

Objectives 

Safety  of  flight 
clearance; 

Safe  escape  maneuver 
determination 

Safety  of  flight 
clearance;  safe  escape 
maneuver 

determination 

Minimum  low- 
altitude  safe  release 
range;  risk 
assessment 

Assumptions 

Launch 

aircraft 

maneuvers 

Assume  straight  and 
level  is  worst  case; 
fixed  “g”  maneuvers; 
altitudes  &  speeds  from 
tactics  guides 

Assume  straight  and 
level  is  worst  case; 
fixed  “g”  maneuvers; 
altitudes  &  speeds 
from  tactics  guides 

Use  Hover,  Bank, 

Dive,  break  turn 
toward  masking 
terrain  after  launch, 
or  vertical  or  lateral 
unmask  &  egress 

Weapon 

variations 

Hot/cold  motor  when 
data  available;  no  roll 
variations;  variable 
launch  modes 

Hot/cold  motor  when 
data  available;  no  roll 
variations;  variable 
launch  modes 

Hot/cold  motor  when 
data  are  available  and 
IFS  fidelity  supports 

S/A  variations 

Spec  value  plus  and 
minus  tolerance 

Spec  value  minus 
tolerance 

Spec  value  minus 
tolerance  +  delay 

UNK 
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Category 

Issue 

NAWCWD 

NAWCAD 

Seek  Eagle 

AMRDEC 

Definitions 

“Safe  Escape  Analysis” 

“Safe  Escape 
Analysis” 

“Safe  Escape 

Analysis” 

“Safe  Escape 

Analysis” 

Aircraft 

Modeling 

Physical 

Description 

6  view  presented  area 

6  view  presented  area 

6-sided  box  enclosing 
aircraft  +  CAD  model 

Vulnerability 

Description 

6  view  vulnerable  area 

NA 

NA 

AJEM  model 

Target 

Maneuvers 

(air-to-air) 

Straight  and  level 
(assumed  worst  case); 
occasionally  consider 
target  maneuvers 

NA 

UNK 

NA 

Aircraft  Flight 
Path  Model 

JAAM 

JAAM,  AWDS 

RCAS,  FlightLab 

Target  Debris 
Model 

Not  modeled 

Not  modeled 

Not  Modeled 

Weapon 

Modeling 

Weapon 

Trajectory 

Program  Office  6-dof 

Program  Office  6-dof 

Program  Office  6-dof 

Motor  Temp 

Use  hot/cold  variations 
if  data  available 

Use  hot/cold 
variations  if  data 
available 

Use  hot/cold 
variations  if  data 
available 

Debris  Pattern 
Source 

Arena  Test  Data 

Arena  Test  Data 

Arena  Test  Data 

Debris  Frag 
Zones 

5-10  deg  polar  zones, 
uniform  distribution 

10  deg  polar  zones; 

24  azimuth  zones 

5  deg  polar  zones, 
uniform  distribution 

Debris  Frags 

Large  frags  modeled 
separately 

Large  frags  modeled 
separately 

UNK 

Warhead  frags  can  be 
modeled  separately 

Warhead  frags  can  be 
modeled  separately 

UNK 

No  min  frag  size 

UNK 

KE>5  ft-lbs 

No  min  frag  velocity 

UNK 

v>v50 
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Category 

Issue 

NAWCWD 

NAWCAD 

Seek  Eagle 

AMRDEC 

Weapon 

Modeling 

Debris  Frags 

No  data  for  statistical 
variations 

No  data  for  statistical 
variations 

Monte  Carlo  frag  fly¬ 
out  simulation 

SA  Device 

Modeled  as  arm  time 
plus  and  minus  spec 
tolerance 

Spec  value  minus 
tolerance 

Spec  value  minus 
tolerance  +  delay 

UNK 

M&S 

SA  M&S 

Used 

ASEP 

Path  4 

CASES 

ASEAT 

Capability 

Adds  asymmetric  roll 
zones  to  Path  3-D 

3-D  dynamic  frag 
zones;  calculates  Ph 
or  Pk 

Pre-generated 
warhead  data  files 
available;  adds  GUI 
to  ASEP 

Monte-Carlo,  two 
passes  (hit  box,  then 
detailed  CAD  model) 

Accuracy 

No  formal  V&V 
available 

No  formal  V&V 
available 

No  Formal  V&V 
available 

Some  V&V  on 

AJEM 

Comparison  runs 
between  ASEP  and 
CASES 

Comparison  runs 
between  ASEP  and 
CASES 

No  V&V  documented 
on  ASEAT 

No  data  V&V 
documentation 

No  data  V&V 
documentation 

No  data  V&V 
documentation 

No  formal  validation 

No  formal  validation 

No  formal  validation 

Accreditation  Package 
done  by  Seek  Eagle 

Accreditation 

Package  done  by 

Seek  Eagle 

Usability 

User  Manual  and 

Analyst  Manual  (Dec 
1996) 

UNK  documentation 

No  documentation  on 
ASEAT 

Seek  Eagle  provides 
limited  user  support 

Seek  Eagle  provides 
limited  user  support 
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APPENDIX  A:  SURVEY  QUESTIONS  &  INFORMATION  CATEGORIES 


Assumptions  and  Requirements 

Requirements 

Phit  only 
Pk  and/or  Phit 

Probability  Requirement  &  rationale 
0.01 
0.0001 

Consider  other  outside  hazards  (from  threat  weapons,  etc) 

If  analysis  is  required  to  justify  S/A  times  that  do  not  meet 
the  probability  requirements,  what  analysis  process  is  used? 

Require  maneuver  after  launch  if  probability  threshold  not  met  in  all  cases? 

What  is  required  in  order  to  pass  the  flight  clearance  "safe  escape  box"? 

What  is  the  objective  of  the  analysis? 

Safe  arm  time/distance  determination? 

Risk  Assessment? 

Safe  Escape  maneuver  determination? 

Safety  of  flight  clearance? 


Assumptions 

Launch  aircraft  maneuvers 

Straight  and  level  flight 

Fixed  "g"  maneuvers  (horizontal,  vertical) 

Find  safe-escape  maneuvers 
Launch/release  altitudes,  speeds 

Assume  only  one  weapon  trajectory  per  release  condition,  or  include  variations? 
Hot/Cold  motor 
Roll  variations/statistics 
Environmental  variations  (wind,  etc.) 

Variations  in  S/A  functioning? 

Fixed  time/distance? 

Statistical  Distribution? 


Definitions 

How  do  you  define  "safe  separation  analysis"? 
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How  do  you  define  "Safe  escape  analysis"? 

How  do  you  define  "safe  arming  analysis"? 

MIL  HDBK  1763  definitions? 

MIL-STD  1316E? 

Fuze  Management  Board  Joint  Agreement? 

Aircraft  Modeling 

Physical  description 

6  view  Presented  Area 
26  View  Presented  Area 
Detailed  model 

Vulnerability  description 

6  view  vulnerable  area 
26  view  vulnerable  area 
Detailed  model  of  components/subsystems 
Source? 


Maneuvers 

Straight  and  level 
Fixed-g  turns 
Complex  flight  paths 
Safe  escape  maneuvers 

Flight  Path  modeling 
Linear 
3-dof 

5- dof 

6- dof 

Target  Debris  model  (after  weapon  impact)? 

Weapon  Modeling 

Trajectory 

Simple  equations 
3-dof 

5- dof 

6- dof 

Motor  temperature 

Debris  Pattern 
Sources 
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Arena  test  data  (static,  dynamic) 

Simulation 
Surrogate  system 
Fragment  zones 

Fragment  zone  sizes 
Fragment  distribution  within  zone 
Average  #  frags  per  zone? 

Distribution  over  zone? 

Fragment  size  distribution  within  zone? 

Fragment  velocity 

Equal  over  zone? 

Fixed  at  zone  interfaces< 

Distribution  over  zone? 

Varies  with  fragment  size? 

Individual  fragments  modeled  (bomb  lugs,  etc)  separately? 

Warhead  fragments  separate  from  debris  fragments? 

Minimum  fragment  size  included  (lethal  fragments) 

Minimum  fragment  velocity  included 
Statistical  variations? 

Safety  and  Arming  device 
Arm  time  only 
Arm  distance  only 
Distance  integrator  model 
Error  distribution 

M&S,  Assumptions 

Source  of  weapon  trajectory  simulation 
Weapon  Developer 

Government  development  or  independent  analysis  shop 
Safe  separation  M&S 

S/A  M&S  Utilized 
Path  3-D 
ASEP 
CASES 
Other 

Significant  Differences  between  S/A  M&S 
Capability 

What  are  the  significant  differences  in  capability  between  the  codes? 

Are  there  differing  levels  of  fidelity? 

If  so,  are  authoritative  data  available  to  support  the  higher  levels  of 
fidelity? 
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Are  functions  modeled  in  one  code  that  are  not  available  in  another? 


Accuracy 

Software  Accuracy 

What  verification  activities  have  been  conducted  and  documented? 
Is  there  a  software  design  specification? 

Have  CASE  tools  been  applied  to  the  code? 

Data  Accuracy 

What  data  V&V  activities  have  been  conducted  and  documented? 
Have  embedded  data  been  V&Y'd? 

Output  Accuracy 

What  validation  activities  have  been  conducted  and  documented? 
Have  there  been  expert  reviews  of  code  input/output? 

Has  the  code  been  benchmarked  against  other  authoritative  codes? 

Usability 

Is  there  an  active  user  group? 

What  documentation  is  available?  Is  it  up  to  date? 

Is  there  support  for  user  questions,  problems? 


29 


Material  Presented  Herein  Is  N 


Technology  Readiness  Assessments: 
Milestone  B  Requirement  for 
Technologies  to  be  Demonstrated 
in  a  Relevant  Environment 


NDIA  10th  Annual  Systems  Engineering  Conference 

October  22-25,  2007 


ID A 
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Outline 


*  Introduction 

-  What  is  a  TRA 

-  Why  it  is  important 

*  Impact  of  10  USC  2366a  on  TRA  processes  and 
practices 

-  Technology  maturation  policy 

-  Best  practices  for 

•  Increasing  TRA  rigor 

•  Focusing  on  technology  maturity  in  source  selection 

•  Using  the  ADM  to  pursue  technology  maturation  and 
insertion  after  Milestone  B 

*  References  and  Resources 


How  TRAs  Got  Started 


“Program  managers’  ability  to  reject  immature  technologies  is 
hampered  by  (1)  untradable  requirements  that  force  acceptance 
of  technologies  despite  their  immaturity  and  (2)  reliance  on 
tools  that  fail  to  alert  the  managers  of  the  high  risks  that  would 
prompt  such  a  rejection.”  GAO/NSIAD-99-162 

“Identify  each  case  in  which  a  major  defense  acquisition  program  entered 
system  development  and  demonstration  ...  into  which  key  technology  has 
been  incorporated  that  does  not  meet  the  technology  maturity  requirement ... 
and  provide  a  justification  for  why  such  key  technology  was  incorporated  and 
identify  any  determination  of  technological  maturity  with  which  the  Deputy 
Under  Secretary  of  Defense  for  Science  and  Technology  did  not  concur  and 
explain  how  the  issue  has  been  resolved.”  National  Defense  Authorization 
Act  for  Fiscal  Year  2002 

“The  management  and  mitigation  of  technology  risk,  which  allows  less  costly 
and  less  time-consuming  systems  development,  is  a  crucial  part  of  overall 
program  management  and  is  especially  relevant  to  meeting  cost  and  schedule 
goals.  Objective  assessment  of  technology  maturity  and  risk  shall  be  a  routine 
aspect  of  DoD  acquisition.”  DoDI  5000.2,  paragraph  3.7. 2.2 


Stop  launching  programs  before  technologies  are  mature 


What  is  a  TRA? 


Systematic,  metrics-based 
process  that  assesses  the 
maturity  of  Critical 
Technology  Elements  (CTEs) 

-  Uses  Technology  Readiness 
Levels  (TRLs)  as  the  metric 

Regulatory  information 
requirement  for  all 
acquisition  programs 

-  Submitted  to  DUSD(S&T)  for 
ACAT  ID  and  1AM  programs 


^  Not  a  risk  assessment 

f  Not  a  design  review 

f  Does  not  address  system 
integration 


Critical  Technology  Element  (CTE)  Defined 


A  technology  element  is  “critical”  if  the  system 
being  acquired  depends  on  this  technology 
element  to  meet  operational  requirements  with 
acceptable  development  cost  and  schedule  and 

with  acceptable  production  and  operation  costs 
and  if  the  technology  element  or  its  application  is 

either  new  or  novel. 

Said  another  way,  an  element  that  is  new  or  novel  or 
being  used  in  a  new  or  novel  way  is  critical  if  it  is 
necessary  to  achieve  the  successful  development 
of  a  system,  its  acquisition  or  its  operational  utility. 

CTEs  may  be  hardware,  software,  manufacturing,  or  life  cycle  related 

at  the  subsystem  or  component  level 


TRL  Overview 


*  Measures  technology  maturity 

*  Indicates  what  has  been  accomplished  in  the 
development  of  a  technology 

-  Theory,  laboratory,  field 

-  Relevant  environment,  operational 
environment 

-  Subscale,  full  scale 

-  Breadboard,  brassboard,  prototype 

-  Reduced  performance,  full 
performance 

*  Does  not  indicate  that  the  technology  is  right  for 
the  job  or  that  application  of  the  technology  will 
result  in  successful  development  of  the  system 
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Increasing  maturit 


Hardware  and  Manufacturing  TRLs 


>i 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 


Basic  principles  observed  and  reported 


Technology  concept  and/or  application 
formulated 

Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept 

Component  and/or  breadboard 
validation  in  a  laboratory  environment 

Component  and/or  breadboard 
validation  in  a  relevant  environment 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 


System  prototype  demonstration  in  an 
operational  environment 

Actual  system  completed  and  qualified 
through  test  and  demonstration 

Actual  system  proven  through 
successful  mission  operations 
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Overview  of  Technology  Considerations 
During  Systems  Acquisition 


Joint  Capabilities 
Integration  & 
Development 
System  (JCIDS) 


User  Needs  & 
Technology  Opportunities 


^  Process  entry  at  Milestones  A,  B,  or  C 

*  Entrance  criteria  met  before  entering  phase 

?  Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

\  Concept 
/  Decision 

J\  Design 
<  >  Readiness 

V  Review 

LRIP/IOT&E  Decision 

V  Review 

t 


Pre-Systems  Acquisition 


ICD 


i 

CCD 


Systems  Acquisition 
CPD 


Sustainment 


TRAs  required  at  MS  B,  MS  C,  and  program 
initiation  for  ships  (usually  MS  A). 
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Why  is  a  TRA  Important?  (i  of  2) 


The  Milestone  Decision  Authority 
(MDA)  uses  the  information  to  support 
a  decision  to  initiate  a  program 

-  Trying  to  apply  immature  technologies 
has  led  to  technical,  schedule,  and  cost 
problems  during  systems  acquisition 

-  TRA  established  as  a  control  to  ensure 
that  critical  technologies  are  mature, 
based  on  what  has  been  accomplished 


Congressional  interest 

-  MDA  must  certify  to  Congress  that 
the  technology  in  programs  has 
been  demonstrated  in  a  relevant 
environment  at  program  initiation 

-  MDA  must  justify  any  waivers  for 
national  security  to  Congress 


Quantifying  the  Effects  of  Immature 

Technologies  <iof2) 


According  to  a  2006  GAO  review  of  52  DoD  programs: 

-  Only  10%  of  programs  began  SDD  with  mature 
technology  (TRL  7) 

•  Programs  that  started  with  mature  technologies  averaged 
4.8%  development  cost  growth  and  a  1%  acquisition  unit 
cost  growth 

•  Programs  that  did  not  have  mature  technologies  averaged 
34.9%  development  cost  growth  and  a  27%  acquisition  unit 
cost  growth 

-  Only  23%  of  programs  began  SDD  with  DoD’s  standard 
for  mature  technology  (TRL  6) 

•  Programs  that  started  with  mature  technologies  averaged 
18.8%  development  cost  growth 

•  Programs  that  started  with  mature  technologies  averaged 
34.6%  development  cost  growth 


Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-06-391 ,  March  2006 


Quantifying  the  Effects  of  Immature 

Technologies  <2  of  2) 

According  to  a  2007  GAO  review  of  33  DoD  programs: 

-  Only  16%  of  programs  began  SDD  with  mature 
technology  (TRL  7) 

•  Programs  that  started  with  mature  technologies  averaged 
2.6%  development  cost  growth,  a  1%  acquisition  unit  cost 
growth,  and  a  1  month  schedule  slippage 

•  Programs  that  did  not  have  mature  technologies  averaged 
32.3%  development  cost  growth,  a  30%  acquisition  unit 
cost  growth,  and  a  20  month  schedule  slippage 

-  Only  32%  of  programs  began  SDD  with  DoD’s  standard 
for  mature  technology  (TRL  6) 

•  Programs  that  started  with  mature  technologies  averaged 
8.4%  development  cost  growth 


Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-07-406SP,  March  2007 
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Why  is  a  TRA  Important?  (2  of  2) 


•  The  PM  uses  the  expertise  of  the  assessment  team  and 
the  rigor  and  discipline  of  the  process  to  allow  for: 

-  Early,  in  depth  review  of  the  conceptual  product  baseline 

-  Periodic  in-depth  reviews  of  maturation  events  documented 
as  verification  criteria  in  an  associated  CTE  maturation  plan 

-  Highlighting  (and  in  some  cases  discovering)  critical 
technologies  and  other  potential  technology  risk  areas  that 
require  management  attention  (and  possibly  additional 
resources) 

•  The  PM,  PEO,  and  CAE  use  the  results  of  the 
assessment  to: 

-  Optimize  the  acquisition  strategy  and  thereby 
increase  the  probability  of  a  successful  outcome 

-  Determine  capabilities  to  be  developed  in  the  next 
increment 

-  Focus  technology  investment 


12 


PM  responsibility 


Process  Overview 


Collect 

data 


>=> 


PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  bUSb(SAT)  informed 

PM  responsibility 
Best  Practice:  Independent 
review  team  appointed  by  SAT 
Exec  verifies 

PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  bUSb(SAT)  informed 

SAT  Exec  responsibility 
Appoints  independent  review 
team  to  do  it;  PM  funds  it 


(  \ 

Coordinate  and  submit  TRA 

\ _ _ J 


SAT  Exec  coordinates 
Acquisition  Executive  submits 


OSD  review 


j 


bUSb(SAT)  responsibility 
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Outline 


*  Introduction 

-  What  is  a  TRA 

-  Why  it  is  important 

*  Impact  of  10  USC  2366a  on  TRA  processes  and 
practices 

-  Technology  maturation  policy 

-  Best  practices  for 

•  Increasing  TRA  rigor 

•  Focusing  on  technology  maturity  in  source  selection 

•  Using  the  ADM  to  pursue  technology  maturation  and 
insertion  after  Milestone  B 
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Technology  Maturation  Policy  Leading  to 
Milestone  B  is  Unambiguous 


“The  project  shall  exit  Technology  Development  when 
an  affordable  increment  of  militarily-useful  capability 
has  been  identified,  the  technology  for  that  increment 
has  been  demonstrated  in  a  relevant  environment,  and  a 
system  can  be  developed  for  production  within  a  short 
timeframe  (normally  less  than  five  years);  or  when  the 

MDA  decides  to  terminate  the  effort . A  Milestone  B 

decision  follows  the  completion  of  Technology 
Development.”  (DoDI  5000.2,  paragraph  3.6.7) 


Technology  Development 


Technology  Maturation  Policy  Leading  to 
Milestone  B  is  Unambiguous  (cont’d) 


“The  management  and 
mitigation  of  technology  risk, 
which  allows  less  costly  and 
less  time-consuming  systems 
development,  is  a  crucial  part  of 
overall  program  management 
and  is  especially  relevant  to 
meeting  cost  and  schedule 
goals. 


Objective  assessment  of  technology 
maturity  and  risk  shall  be  a  routine  aspect 
of  DoD  acquisition.  Technology  developed 
in  S&T  or  procured  from  industry  or  other 
sources  shall  have  been  demonstrated  in  a 
relevant  environment  or,  preferably,  in  an 
operational  environment  to  be  considered 
mature  enough  to  use  for  product 
development  in  systems  integration. 
Technology  readiness  assessments,  and 
where  necessary,  independent 
assessments,  shall  be  conducted. 

If  technology  is  not  mature,  the 
DoD  Component  shall  use 
alternative  technology  that  is 
mature  and  that  can  meet  the 
user’s  needs.”  (DoDI  5000.2, 
paragraph  3.7.2.2) 


and 
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The  Policy  is  Reflected  as  a  Title  10 
Requirement  for  Certification 


10  USC  §2366a  states 

Major  defense  acquisition  programs:  certification 
required  before  Milestone  B  or  Key  Decision  Point 
B  approval: 

(a)  CERTIFICATION.  A  major  defense  acquisition 
program  may  not  receive  Milestone  B  approval, 
or  Key  Decision  Point  B  approval  in  the  case  of 
a  space  program,  until  the  milestone  decision 
authority  certifies  that  - 

(2)  the  technology  in  the  program  has  been 

demonstrated  in  a  relevant  environment; 

•  •  • 

(10)  the  program  complies  with  all  relevant  policies, 
regulations  and  directives  of  the  Department  of 
Defense 


'THE  UNDER  SECRETARY  0r  DEFENSE 
3010  DEFENSE  PENTAGON 
WASHINGTON,  DC  203(11-3310 


UAYOZZDM 


MEMORANE'JM  FOR;  SEE  DISTPJBUTIQK 


SUBJECT:  Implemertatior  of  SEcrian  2J£da  of  Tilts  10,  Unites  Stales  Code 


Section  23£6a  of  tile  10,  United  Slades  Code,  as  enacted  'uy  section  £01  Df  the  Saldana! 
Defense  AnfluHizaton  And  far  Fiscal  Year  2MlS  (Pub.  L.  Xo.  109-163),  requires  the  Milestone 
Decision  Authority  (MDA)  fn:  a  Major  Defense  Acquisition  Program  (MDAP)  Id  maia 
certain  certifications  prim  to  Milestone  3  or  Key  Decision  Point  B  approval 

To  fidfiL  this  requirement  .dbe  MDA,  without  the  authority  Id  delegate,  shall  sign  a 
memorandum,  subject  'Pro-gram  Certification, 1  prior  1o  signing  die  Acquisition  Decision 
Memoracdum  (ADM).  This  certification  memorandum  shall  be  prepared  'for  rhe  record," 
and  shall  include  ibe  staiemects  in  tbe  attachment  without  modification.  I:  die  program  is 
initiated  at  a  latar  derision  point  e.g..  Milestone  C,  a  si  rrlar  memorandum  shaLhe 
prepared,  as  a  nianer  of  policy,  consistent  nidi  the  indent  of  the  slatate. .  The  certification 
memorandum  shaL  be  submitted  to  the  congressional  defense  cammitiees.  as  defined  at  10  U. 
5.C.  1Q1_(16)_  with  tbe  ficst  Sal  e  c  Dei  Ac  qui  siton  Report  for  the  program  after  cmnpteton  of 
the  certification. 

The  MDA  may  waive  one  cc  rcora  o-f  components  (!)  through  (6)  of  the  required 
certficalian  (specifically,  one  or  more  ofpiaragraphs  (1)  through  (£)  in  the  attachment)  foe  an 
MDAPiflbe  MDA  determines  than  but  foe  such  a  w  aiver,  she  Department  would  be  ucable 
1o  meel  critical  national  security  objectives.  The  MBA  shall  submit  the  waiver,  the 
deter  mean  don.  and  reasons  for  the  determination,  in  writing,  ro  the  congressionaL  defense 
committees  within  JC  days  of  authorizing  Ibe  waiver.  Tbe  MDA  may  not  delegate  this 
waiver  authority. 

In  addition  tD  tbe  certification  memorandum,  tba  MDA  will  include  tbe  folio-wing 
statement  in  the  ADM:  '!  hat  e  reviewed  the  program  and  have  made  ibe  certifications 
required,  or  executed  a  waiver  as  authorized,  by  secton  BGSa  of  lille  10,  United  Slates 
Code. 1 


This  policy  shall  apply  to-  ME-APs  approve d  tv  me  and  Id  MD.APs  managed  by 
Department  of  Defense  Component:  Acquisition  Eneoutves  or  the  Assistant  Secretary 
of  Defense  for  Networks  anilnfoimii.iinn  btepation.  Hus  requirement  went  btc  effect 
January  6.  2006,  and  dull  be  reflected  m  the  nesl  revision  to  Department  o-f  Defense 
Instructoc  5CHJ0.2. 


Attachment: 
As  stated 


Certification  Submitted  with  the  First  Selected 
Acquisition  Report  for  the  Program 
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•  •  •  But  Waivers  Are  Allowed 


(c)  WAIVER  FOR  NATIONAL  SECURITY.  The 
milestone  decision  authority  may  waive  the 
applicability  to  a  major  defense  acquisition 
program  of  one  or  more  components  (as  specified 
in  paragraph  ( 1 ),  (2),  (3),  (4),  (5),  ( 6 ),  (7),  (8),  or  (9)  of 
subsection  (a))  of  the  certification  requirement  if 
the  milestone  decision  authority  determines  that , 
but  for  such  a  waiver ,  the  Department  would  be 
unable  to  meet  critical  national  security  objectives. 

Whenever  the  milestone  decision  authority  makes  such  a 
determination  and  authorizes  such  a  waiver,  the  waiver,  the 
determination,  and  the  reasons  for  the  determination  shall  be 
submitted  in  writing  to  the  congressional  defense  committees 
within  30  days  after  the  waiver  is  authorized. 
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Changes  Made  to  Meet  the  Statutory 

Requirements 


USD(AT&L)  practice 

-  Programs  will  no  longer  be  initiated  with 
immature  technologies 

-  The  same  standards  apply  to  all 
acquisition  programs 

DDR&E  will  provide  technical  advice  to  the 

MDA  in  support  of  certification 

-  For  MDAPs,  MAIS,  and  space  systems 

-  Approved  TRA  process  and  report  will 
be  the  basis  of  that  advice 


19 


Draft  New  5000.2  Language 
September  13,  2007 


*  For  the  SDD  Phase: 

3. 6. 7.1  “Final  Requests  for  Proposals  (RFPs)  for  the  SDD 
phase  shall  not  be  released,  nor  shall  any  action  be  taken 
that  would  commit  the  program  to  a  particular  contracting 
strategy  for  SDD,  until  the  MD A  has  approved  the 

Acquisition  strategy.  The  PM  shall  include  language 
in  the  RFP  advising  offerors  that  (1)  the 
government  will  not  award  a  contract  to  an 
offeror  whose  proposal  is  based  on  technologies 
that  have  not  been  demonstrated  in  a  relevant 
environment,  and  (2)  that  offerors  will  be 
required  to  specify  the  technology  readiness 
level  of  the  key  technologies  on  which  their 
proposal  is  based  and  to  provide  reports 
documenting  how  those  technologies  have  been 
demonstrated  in  a  relevant  environment." 


20 


Prototyping  and  Competition  Policy 
Promotes  Technology  Maturity 


MUitfribASIHlU  IC*  -ui  b  l.iHII  fffH  Fill  ^111 1  III  Md  i  «  S  I* 
ifUUM^or  hif  U^TOVia  STatfl 
COMUMIJL  Li  1  UttlU  0?lUiniMCOUMA 

tmTOHef  T>*  BCFI^E  .VLftMH* 

iiaen  *&vi*t>**Qm*&* 


Policy 

-  During  SDD,  large  teams  should  be  producing  detailed 
manufacturing  designs  -  not  solving  myriad  technical 
issues 

-  All  pending  and  future  programs  formulate  acquisition 
strategies  and  provide  funding  for  two  or  more 
competing  teams  to  produce  prototypes  of  key  system 
elements  through  Milestone  B 

Promotes  maturity  via 

-  More  rigorous  relevant  environment  demonstrations 

-  More  comprehensive  evidence  of  maturity 

-  Fewer  technical  problems  in  the  final  design 

-  Using  prototypes  for  accelerated  life-cycle  tests 

-  Providing  insight  into  production  issues 
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Desired  Outcomes  from  Process  Changes  in 
Support  of  the  New  Situation 


•  Safeguards  in  place  to  provide  the  DDR&E  with  the 
confidence  necessary  to  assure  the  MDA  that 
certification  can  be  made 

-  To  make  the  TRA  support  the  certification,  it  must 
draw  upon  the  best  technical  information  available 
prior  to  source  selection 

•  Assurance  that  technologies  have  been 
demonstrated  in  a  relevant  environment  by  the 
winning  SDD  Phase  contractor 

-  To  initiate  programs  with  mature  technologies,  the  source  selection 
process  should  include  a  focus  on  technical  maturity 

•  ADM  language  establishing  conditions  for  CTE  insertion  after 
Milestone  B 

-  To  initiate  programs  with  mature  technologies,  immature  CTEs  may 
be  pursued  in  a  parallel  development  effort,  if  approved  maturation 
plans  submitted  with  the  TRA —  on  ramp  vice  offramp  for  preferred 
approaches  with  undemonstrated  technologies 


Processes  and  Best  Practices  Must  Change 
to  Achieve  These  Outcomes  <i  of  2) 


-  Nevertheless,  many  of  the  changes  to 
increase  rigor  in  the  process  are  indeed 
best  practices 

-  The  new  practices  and  processes  are 
subject  to  change  as  more  experience  is 
gained 

*  All  practices  and  processes  presented 
herein  are  not  official  until  captured  in 
the  TRA  Deskbook 


•  10  USC  2366a  is  new,  so  technically  what 
follows  are  to  some  extent  best  ideas 
vice  best  practices 
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Processes  and  Best  Practices  Must  Change 
to  Achieve  These  Outcomes  <2  of 2) 

a  r 

Increasing  the  rigor  of  the  TRA  process  and 
report  to  assure  a  quality  product 

-  Initiating  the  TRA  process 

-  Identifying  CTEs 

-  Assessing  CTE  maturity 

-  Preparing  the  TRA  report 

Including  a  focus  on  technical  maturity  in  source  selection  to 
assure  mature  technologies  in  programs 

-  Preparing  the  RFP 

-  Assessing  CTE  maturity  during  source  selection 

Using  the  ADM  to  pursue  technology  maturation  and  insertion 
after  Milestone  B 

-  Preparing  maturation  plans  for  CTEs  the  PM  would  like  to  insert 

-  Drafting  ADM  language 


Best  Practices  for  Initiating  the  TRA  Process  <i  of  2) 


_ •  Keep  the  Component  S&T  Executive  and 

v>  Chanel-  the  DUSD(S&  informed  throughout  the 
l  Deskbook  J  entire  TRA  process 

•.Begin  the  TRA  process  12-14  months  prior 
Nochange  |  to  the  anticipated  Milestone/KDP  B  for  an 
Deskbook  J  ACAT  ID  or  1AM  program 

-  Component  should  immediately  prepare  a 
schedule 

-  Align  with  the  Acquisition  Strategy 

-  Incorporate  into  IMS 


-  DUSD(S&T)  should  provide  comments  on 
the  schedule  and  indicate  conditions  for 
agreement 
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Best  Practices  for  Initiating  the  TRA  Process  (2  of 2) 


>/  Establish  an  Independent  Panel  of 
AtidsriS.,r  |  subject  matter  experts  to  identify 

candidate  CTEs  and  assess  CTE 
maturity 

-  Must  be  genuine  subject  matter  experts 
in  relevant  disciplines 

-  Must  be  independent  of  program  to  be 
credible 

-  Coordinate  with  DUSD  (S&T)  who 
should  provide  comments  and  indicate 
conditions  for  agreement 
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Best  Practices  for  Identifying  CTEs 


current 

Deskbook; 

shifts 

responsibility 
from  the  PM 


Use  the  Independent  Panel  of  subject  matter  experts  to 
identify  CTE  candidates  in  conjunction  with  the  program; 
final  list  should  reflect  DUSD(S&T)  recommended  changes 

-  Most  current  system  design  should  be  the  starting  point  for 

-  Any  system/component  may  be  a  CTE  until  proven  otherwise 


V 


to  the  Panel 


J 


Determination  is  based  on  system  design,  WBS,  or 
architecture  provided  by  the  program 

PM  should  prepare  an  initial  list  of  CTE  candidates 

•  Based  on  information  from  BAAs,  RFIs,  and  literature  searches 
along  with  actual  results  from  program-funded  efforts  in 
government  laboratories  or  industry  when  there  will  be  a  full  and 
open  competition  with  a  formal  source  selection  process 

•  Based  on  planned  technical  approaches  (e.g.,  actual  design  when 
available  otherwise  use  government’s  reference  architecture) 
when  sole  source  or  competitive  down  select 

Final  CTE  list  established  during  coordination;  DUSD(S&T) 
may  recommend  changes 


•  Additions  may  include  any  technologies  that  warrant  the  rigor  of 
the  formal  TRA  process 
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Best  Practices  for  Assessing  CTE  Maturity 


Adds 
rigor  to 
current 
Deskbook 

V  J 


Use  the  Independent  Panel  to  assess  CTE  maturity;  base  all 
conclusions  on  objective  evidence  and  the  technical 
expertise  of  the  Panel 


Use  the  requirements,  identified  capabilities,  system 
architecture,  and  the  concept  of  operations  and/or  the  concept 
of  employment  to  define  relevant  and  operational 
environments  I 


Form  subteams  (on  the  basis  of  subject 
matter  expertise)  to  deliberate  on  the 
appropriate  TRL  and  then  defend  their 
position  to  the  entire  Panel 

Do  not  constrain  the  assessment  process  to  be  a  response  to  a 
“program-developed”  position  on  the  TRL 
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Best  Practices  for  Preparing  the  TRA  Report 


Adds 

rigor  to  ( 
current 
^  Deskbook  ^ 


The  TRA  report  should  consist  of  (1)  a  short  description  of  the 
program  (2)  the  Independent  Panel  credentials  (3)  Independent 
Panel  findings  and  supporting  evidence  (4)  other  technical 
information  deemed  pertinent  by  the  Component  S&T  Executive 
and  (5)  a  cover  letter  signed  by  the  Component  S&T  Executive 

-  The  Independent  Panel  (Chair)  should  prepare  the  following: 

•  Panel  deliberations,  findings,  and  conclusions 

•  Evidence  and  rationale  for  the  final 
assessment  in  a  way  that  the  chain  of  logic  is 
clear 

-  Reference  specific  sources  using  actual 
pages  in  reports  for  the  evidence 
presented 

-  A  PM’s  self  assessment  of  TRLs  may  be 
included  in  the  cover  letter 

-  The  Component  S&T  Executive  should 
indicate  agreements  or  disagreements  with 
the  findings  of  the  Independent  Panel  in  the 
cover  letter 
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Best  Practices  for  Preparing  the  RFP 


Change  to 
current 
Deskbook 


Include  language  in  the  RFP  to  convey  DoD’s  desire 
to  have  technologies  demonstrated  in  a  relevant 
environment  by  the  winning  SDD  Phase  contractor 

-  A  description  of  the  relevant  environment  and  the 
candidate  CTEs 


-  Based  upon  the  definitions  and  guidance  provided  in 
the  TRA  Deskbook,  offerors  must  identify  all  CTEs  in 
their  proposal,  provide  the  rationale  for  classifying 
them  and  as  CTEs,  assess  their  TRL,  and  include 
sufficient  technical  evidence  to  support  the  TRL 
assessment 

-  A  description  of  how  the  CTEs  and  the  associated 
evidence  of  maturity  will  be  evaluated 
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Other 
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Outline 


*  Introduction 

-  What  is  a  TRA 

-  Why  it  is  important 

*  Impact  of  10  USC  2366a  on  TRA  processes  and 
practices 

-  Technology  maturation  policy 

-  Best  practices  for 

•  Increasing  TRA  rigor 

•  Focusing  on  technology  maturity  in  source  selection 

•  Using  the  ADM  to  pursue  technology  maturation  and 
_ insertion  after  Milestone  B 

*  References  and  Resources 
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References  and  Resources 


Defense  Acquisition  Resource  Center  http://akss.dau.mil/darc/darc.html 

-  DoD  Directive  5000.1  (DoDD  5000.1),  The  Defense  Acquisition  System ,  dated 
May  12,  2003 

-  DoD  Instruction  5000.2  (DoDI  5000.2),  Operation  of  the  Defense  Acquisition 
System,  dated  May  12,  2003 

-  Defense  Acquisition  Guidebook 

DAU  Continuous  Learning  Module  CLE021 

-  https://learn.dau.mil/html/clc/Clc.jsp  to  browse  it 
TRA  Deskbook 

-  http://www.defenselink.mil/ddre/doc/tra_deskbook_2005.pdj 

DDR&E  ^ ^ 

-  Mr.  Jack  Taylor  jack.taylor@osd.mil 

Institute  for  Defense  Analyses 

-  Dr.  Dave  Sparrow  dsparrow@ida.org 

-  Dr.  Jay  Mandelbaum  jmandelb@ida.org 

-  Dr.  Michael  May  mmay@ida.org 
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Revisiting  Aircraft  Flight 
Simulator  Acceptance  Criteria 


999 


1930  2000 

Dean  Carico 

Rotary  Wing  Ship  Suitability 
Naval  Air  Warfare  Center  Aircraft  Division 
Patuxent  River,  MD.,  USA 


Presentation  Outline 


Flight  Training  Background 
Early  Simulation  Fidelity  Studies 
Existing  Simulator  Criteria 
US  Navy  Simulation  Initiative 
Modeling  Mission  Applications 
MFOQA 
Summary 


Lilienthal  Glider  -1894 


Wright  Brothers  First  Flight  1903 


Curtis  JN-4  Barnstorming  ~  1919 


Link’s  Blue  Box  Trainer 


F-18  OFT  Link  “SimuSphere”  design  ~  2001 

External  View  Point 


F-18C  OFT  -  Pilot  View 


MH-60R  Motion  OFT  Layout 


MFS  MH-60R  OFT  #2 


Early  US  Navy  Simulator  Experience 


•  Early  US  Navy  flight  simulator  evaluated  by  fleet 
pilots 

•  NATC  started  evaluating  OFT/WST  with  flight 
test  teams  using  flight  test  data  in  the  1970’s 

-  SH-2F  Device  2F106  -  First  WST  tested  by  test  team 

•  Trainer  acceptance  criteria  developed  by 
NAWCTSD 

•  Criteria  featured  as  OFT/WST  built-in-test 
options 


Early  US  Army  Simulator  Experience 

Considerable  simulator  fidelity  work  done  by 
US  Army/NASA  Ames  in  1980’s  &1990’s 

-  NASA  Ames  Vertical  Motion  Simulator  (VMS) 

-  US  Army  Aeronautical  Design  Standard  33 

•  Handling  qualities  focus 

1984  UH-60  NOE  Simulation 

-  Aircraft  HQR’s  1 .5  to  2  better  than  simulator 

Low  visibility  simulation  for  hover  &  low  speed 

-  Fine-grained  micro  texture  more  important  than 
FOV 


OFT/WST  Acceptance  Criteria 

US  Navy 

-  NAWCTSD  Guidelines  Rotary  Wing  Aircraft 

-  NAWCTSD  Guidelines  Fixed  Wing  Aircraft 

Federal  Aviation  Agency  (USA) 

-AC  120-63  Helicopter  Simulator  Qualification 
-AC  120-40  Airplane  Simulator  Qualification 

Joint  Aviation  Authorities  (Europe) 

-  JAR-STD  1H  Helicopter  Flight  Simulators 
-JAR-STD  1 A  Airplane  Flight  Simulators 


Sample  Helicopter  Simulator  Test  Criteria 


Sample  Test 

NAWCTSD 

FAA  AC  No: 
120-63 

JAR-STD  1 H 

Performance 

Hover  Torque 

Low  A/S  CP 

3% 

2%  full  travel 

+/-  3% 

+/-  5% 

+/-  3% 

+/-  5% 

Handling  Qualities 
Trimmed  CP 

Critical  Azimuth  CP 

2%  full  travel 
2%  full  travel 

+/-  5% 

+/-  5% 

+/-  5% 

+/-  5% 

Autorotation 

Rotor  Speed 

i% 

+/- 1 .5% 

+/- 1 .5% 

Notation:  CP  -  Control  Position 
A/S  -  Airspeed 


On-Going  U.S.  Navy 

Small  Business  Innovative  Research  Project 

•  Topic  N07-033  -  Advanced  Aircraft  Simulator 
Flight  Fidelity  Evaluation  Measures 

•  Four  six-month  Phase  I  contract  awards  made  in 
early  summer  2007 

•  Hopefully,  two  two-year  Phase  II  contracts  will  be 
awarded  following  the  Phase  I  work 

•  Research  topic  endorsed  by  NAVAIR  PMA-205 
Aviation  Training  Systems 

•  NAVAIR  PMA-205  manages  over  3  dozen 
OFT/WST,  plus  numerous  other  training  devices 


Simulator  Project  Focus  Areas 

Aircraft  type  (helicopter,  tilt  rotor,  VTOL,  fixed-wing) 
Flight  Test  Maneuver  (USNTPS,  ADS-33,  Operational) 
Maneuver  Aggressiveness  (timed  non  repeatable  tasks) 
Control  strategy  (workload  and  bandwidth) 

Visual  scene  (fine  grain  texture,  resolution,  FOV,  etc) 
Simulator  component  (visual,  motion,  aural,  cockpit,  etc) 
Time  or  frequency  domain  analysis  options 
Cost  and  risk  (VV&A  using  risk/benefit  analysis) 

Pilot  factor  (flight  time/type/conditions,  attitude) 

Mission  task  element  (Navy,  Army,  Air  Force  missions) 
Total  simulator  system  fidelity  (quantify  cueing  effects) 
Military  flight  operations  quality  assurance  (MFOQA) 


Potential  Virtual  Mission  Applications 

•  Air-to-Air  Combat 

•  Air-to-Air  Refueling 

•  Nap-of-the-Earth 

•  Rotorcraft/Ship  or  Dynamic  Interface 

-  Rotorcraft 
-Ship 

-  Airwake 

-  Ship  motion 
-Visual  environment 


Rotorcraft/Ship  Simulation 

NAVTOLAND  H-2  VMS  Project  1982 

-  Inadequate  FOV  -  forward  landing  circle 

-  Lack  of  texture  -  ship  deck,  hangar  &  sea 

•  Caused  pilots  to  rely  excessively  on  HUD 

•  HUD  was  considered  ~  +  2  HQR  points  for  task 

•  Tried  to  used  “ice  floes”  to  provide  ocean  texture 

-  Airwake  model  -  excessive  in  magnitude  & 
frequency 

The  helicopter/ship  landing  technology  was 
just  getting  started  at  this  time 


TQUTD 


Joint  Shipboard  Helicopter  Integration 
Process  (JSHIP) 

-  Joint  Test  &  Evaluation  Force 

-  Sponsored  by  OSD  Deputy  Director, 
Developmental  Test  &  Evaluation 

-  Purpose  to  Increase  the  Interoperability  of 
Joint  Shipboard  Helicopter  Operations 
(Army/Air  Force  Helicopters  with  Navy 
Ships) 


JSHIP-1 


Dynamic  Interface  Modeling  &  Simulation  System  (DIMSS) 


•  Define  a  Process  for 

Establishing  a  WOD  Flight 
Envelope  Using  Modeling 
and  Simulation 


Demonstrate  that  the  Process 
Works  by  Validating  a  WOD 
Flight  Envelope  Established 
by  Simulation  for  the 
LHA/UH-60A 
Ship/Helicopter  Combination 

-  Use  Vertical  Motion  Simulator 
at  NASA  Ames 

-  Validate  simulation  against 
flight  test  data 

-  Investigate  minimum  cueing 
requirements  (visual,  aural  & 
motion) 


JSHIP-2 


JSHIP  Simulation  Results 


None  of  the  simulator  configurations  tested  achieved  similar 
subjective  ratings  as  in  the  aircraft 

The  simulation  results  could  not  be  used  to  generate 
operational  flight  envelopes 


Military  Flight  Operations  Quality 
Assurance  (MFOQA) 

OSD  focus  since  2005  to  help  reduce 
overall  mishap  rate 

Builds  on  commercial  aviation  practices 
using  operational  trend  analysis  of 
enhanced  flight  data  to  better  identify 
hazards  and  improve  risk  management 

Goal  for  simulator  full  MFOQA  is  2009 

Initial  aircraft  demo  conducted  at  HSL-41 
in  early  2006  using  H-60  with  JAHUMS 


MFOQA 

Militarj'  Flight  Operations  Quality  Assurance 


Fleet  Operational  Readiness  Improvement 
Proactive  Hazard  Identification  and  Elimination 


MFOQA 

Military  Flight  Operations  Quality  Assurance 

A  knowledge  management  process  that  uses 
downloaded  flight  data  after  every  flight 

to  provide 

aircrew,  the  squadron  and  the  Fleet  with 

quantitative  information 

regarding  aircrew  and  aircraft  performance 

to  improve 

training ,  operational  readiness  and  safety. 


Summary 

Pilots  have  used  simulators  since  the  1930’s 

Trainer  acceptance  criteria  same  since  1970’s 

Questions  on  comparing  FT  &  simulator  data 

Large  simulator  technology  database 

US  Navy  initiated  a  program  in  2007  to  develop 
advanced  simulator  flight  fidelity  measures  to 
enhance  flight  trainer  acceptance  criteria 

The  program  team  members  will  be  searching  for 
related  reference  material  during  the  project  life 

Improved  virtual  models  will  also  have  a  positive 
impact  for  future  flight  test  productivity  &  safety 


Next  Generation  Air  Transportation  System: 
Meeting  the  Enterprise  System  Engineering 

Challenges 

Gerald  Friedman  (MITRE) 
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Flying  the  Crowded  Skies:  Challenges  for  Aviation 


TODAY 


Updated  1/26/2007  2:35  PM  ET 


Airline  Delays  Set  Record  in  2006 


Large  airports2  with  the  highest  percentage  of 
delayed  flights  January-November  2006 


By  Alan  Levin, 


Chicago^  O  Hare 
27% 


Boston's  Logan 
26% 


J.  Emilio  Flores  for  The  New  York  Times 

Air  traffic  controllers  at  the  Los  Angeles  airport. 


By  MATTHEW  L.  WALD 
Published:  January  15,  2007 


WASHINGTON,  Jan.  14  —  By  2025,  government  experts  say,  America’s 
three  times  as  many  planes,  and  not  just  the  kind  of  traffic  flying  today.  T1 
of  tiny  jets,  seating  six  or  fewer,  at  airliner  altitudes,  competing  for  space  \ 
drones  that  need  help  avoiding  midair  collisions,  and  with  commercially  o] 
carrying  satellites  and  tourists  into  space. 
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JFK  International  - 
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Airline  delays  inc 
to  record  highs  b 
weather  starting 
airports  and  strai 
passengers,  acc 
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HOT  TOPIC 


Newark  Liberty  - 

International 

33.5% 


Why  the  Skies  Have  Gotten  Crowded 


By  SARAH  NAS  SAUER 


Data  this  past  week  validated  what  many  fliers  already  suspected  —  the  number  of  delays  and 
cancellations  in  June  may  have  been  the  worst  ever.  According  to  FlightStats.com,  20,301  flights 
were  canceled  in  the  U.S.,  more  then  double  the  number  grounded  in  June  2006.  Among  the  40 
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Congress  threat* 
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Record  Delays  Loom 

Summertime  and  the  traveling  won't  be 
easy  in  the  U.S.  National  Airspace  System 


David  Hughes/Herndon,  VA. 


With  summer  air  traffic  delays  expected  to  set  a  record  for  the  second  year  in  a  row,  the 
FAA  acknowledges  that  long-term  relief  will  depend  on  the  not-yet- assured  NextGen 
ATC  system. 
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THE  MIDDLE  SEAT 
By  SCOTT  MCCARTNEY 


Why  Fliers  Find 
Summer  Travel 


Growing  Tougher 


James 

an 

i  lion's 


Number  of  Cancellations 
More  Than  Doubles 
Amid  Storms,  Gridlock 


1 2-1 8 


Pressure  from  Delays  and  Cancellations 
on  Airline  Quality  of  Service  Standards 


al)c  hinstpntjtcm  Post 


Flying  Late,  Arriving  Light 


Air  Carriers  Are  Delaying  More  Flights  and  Losing  Your  Shirts 


The  Atlanta  Journal-Constitution 


Congress  eyes  standards  for  airline  service 


By  BOB  DART 

The  Atlanta  Journal-Constitution 
Published  on:  04/20/07 


Washington  —  Airlines  have  not  kept  their  promises  to  protect  passengers  f 
horrors,  so  Congress  may  need  to  set  federal  standards  for  customer  servic 
Transportation  Department  investigator  told  a  House  subcommittee  Friday. 


(Ebc  JscUt  jlork  (times 

Federal  Agency  Investigating  Airline  Arrival-Time  Promises 


(By  Lairence  Kesterscn  -  Associated  Press 


By  Del  Quentin  Wilber 
Washington  Post  Staff  Writer 

Thursday,  February  8,  2007;  Page  D01 


Air  travelers  had  a  tough  year  getting 

Airlines'  on-time  performance  droppe 
arriving  late  or  not  at  all,  according  tc 
Statistics. 


It  was  the  worst  year  since  2000,  whe 
from  getting  to  their  destinations  on  t: 

The  airlines  also  mishandled  a  massi\ 
1,000  passengers,  the  industry's  worst 
in  lost  bags  stemmed  from  stricter  sec 
their  luggage. 


There  was  less  consensus  on  the  incre 
and  that's  one  of  the  reasons  2006  hit 
Federal  Aviation  Administration,  saic 


By  JEFF  BAILEY 
Published:  April  21 , 2007 


The  Transportation  Department  said  yesterday  that  it  was  investigating  several 
domestic  airlines  for  publishing  unrealistic  flight  schedules  —  including  ones  that  list 
arrival  times  the  carriers  know  they  cannot  achieve  —  and  said  as  many  as  eight  could  be 
fined  for  failing  to  provide  accurate  flight -delay  information. 


The  agency  is  under  pressure  from  Cc 
recent  episodes  where  several  carrier 
Airways,  stranded  passengers  for  hoi 


The  actions  disclosed  Friday  do  not  d 
which  is  being  reviewed  by  the  agenc; 
after  June  30. 


Rather,  these  actions  aim  at  airlines  t 
to  customers,  when  asked,  the  on-tin 
required. 


A  Transportation  Department  spoke: 
not  to  be  identified,  said  the  agency  c 
information  on  particular  flights.  The 
statistic  on  41  percent  of  the  calls,  the 
involved. 
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^  THE  MIDDLE  SEAT 


By  SCOTT  MCCARTNEY 


A  Report  Card 

On  the  Nation’s  Airlines 


Despite  Financial  Recovery^ 
Many  Carriers  Still  Plagued 
By  Spotty  Customer  Service 


1  10th  congress 
1st  Session 


H.R.  1303 


To  amend  title  49,  United  States  Code,  to  improve  air  carrier  passenger 
services. 


A  BILL 


To  amend  title  49,  United  States  Code,  to  improve  air  carrier 
passenger  services. 

1  Be  it  enacted  by  the  Senate  and  House  of  Represent a- 

2  fives  of  the  United  States  of  America  in  Congress  assembled , 

3  SECTION  1.  SHORT  TITLE. 

4  This  Act  may  be  cited  as  the  “Airline  Passenger  Bill 


5  of  Rights  Act  of  2007”. 
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Pressure  from  Airlines  on 
ATM  System  Performance 


Money. 
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Delta  Takes  Steps  to  Mitigate  Impact  On  Customers 
as  Severe  Weather  Approaches  Northeast 

Delta  Urges  Congress  to  Modernize  Air  Traffic  Control  System 


June  28,  2007:  12:03  PM  EST 


ATLANTA,  June  28,  2007  (PR 
customers  booked  on  flights 
U.S.  to  make  adjustments  to 
weather  expected  in  the  regi< 
the  Northeast  corridor  and  af 
flights  could  be  canceled.  Cu: 
cancellations  may  request  re 
without  fee  or  penalty. 

Impacted  cities  include  the  fc 

New  York  (JFK) 

New  York  (LGA) 

Newark 
Hartford 
Providence 
Boston 

Washington  Reagan 
Washington  Dulles 
Baltimore 
Philadelphia 
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Gridlock  in  the  Air 

No  one  who  has  traveled  by  plane  recently  needs  to  be  told  that  our  commercial  air-travel  system 
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Airlines  Attribute  Delays  to  Poor  Traffic  Control 

April  3,  2007  |  5:15  PM  ET  |  Permanent  Link 

In  response  to  an  Airline  Quality  Rating  report  released  yesterday  that  found  that 
instances  of  delayed  flights  and  lost  baggage  were  up  last  year  over  2005,  the  Air 
Transport  Association,  the  trade  group  representing  the  major  airlines,  released  a 
statement  yesterday. 

In  short,  the  airlines  attribute  the  uptick  in  delays  to  one  of  the  most  contentious 
political  issues  in  transportation  this  year:  the  updating  of  the  air  traffic  control  system 
and  just  who  should  pay  the  bulk  of  the  bill  for  the  much -needed  improvements. 

"The  2007  Airline  Quality  Rating  study  once  again  focuses  on  the  symptoms  rather  than 
the  root  causes  of  passenger  and  airline  frustrations,"  ATA  President  and  CEO  Jim  May 
notes  in  the  statement.  He  says  that  since  the  majority  of  delays  are  caused  by  weather 
problems  that  the  current  system  can't  handle,  Congress  must  now  take  its  "historic 
opportunity"  to  "approve  an  action  plan  for  the  Next  Generation  Air  Traffic  Control 
system  ...  while  ending  the  multibillion-dollar  subsidy  of  business  jets  at  the  expense  of 
the  commercial  airline  passengers." 

For  the  inside  story  on  the  big-time  Washington  battle  over  restructuring  the  FAA,  see 
associate  editor  Angie  C.  Marek's  March  30,  2007,  article  here.  Marek  tells  us  that  bills 
addressing  the  subject  should  appear  in  Congress  soon. 


"Delta's  focus  is  always  the  c 
continue  to  work  to  mitigate 
delays.  However,  today's  sto 
Congress  act  to  modernize  tl 
Kolshak,  Delta's  executive  vi< 
fundamentally  unfair  to  our  customers  that  we  are  operating  in  a  system  that  wasi 
built  in  the  1940s  and  can't  accommodate  today's  air  travel  demand  without  costly 
and  frustrating  delays  and  congestion  that  are  beyond  our  control.  The  FAA  has 
presented  a  plan  to  Congress  that  helps  ensure  airline  passengers  are  provided 
with  an  updated,  21st  century  air  traffic  control  system.  We  urge  Congress  to 
approve  the  FAA's  plan  to  increase  airspace  capacity,  especially  in  the  Northeast, 
and  to  get  away  from  the  status  quo  and  act  boldly  to  modernize  our  nation's 
outdated  air  traffic  control  system." 


:atistics  support  the  anecdotal  evidence  of  crowded  airspace,  taxi  ways  and 
linistrator  Marion  Blakey  recently  noted  that  2006  was  the  worst  year  on 
M  cancellations  and  that  2007  bids  to  be  worse  still.  On  one  day  this  past 
Ilf  the  planes  at  JFK  in  New  York  City  were  on  time.  Nationwide  for  the 
re  than  30%  of  all  flights  were  delayed.  It's  enough  to  make  you  think  they 
K>ur  to  every  departure  time. 

f  course.  But  we're  not  sure  that  firing  Ms.  Blakev  —  which  is  Chuck 
rm  for  addressing  the  problem  —  would  be  any  more  effective.  Ms.  Blakey 
Lg  anyone  who  will  listen  to  fix  some  of  the  problems  with  the  air-travel 
>efore  this  summer's  delays  put  the  overcrowding  in  the  headlines  and  on 


lough  with  the  air-traffic  controllers  union,  which  has  placed  her  on  the  least- 
liof  a  number  of  Democratic  politicians,  including  New  York's  Senator 
dly,  Ms.  Blftkey  last  year  fought  and  won  a  battle  with  the  controllers  union 
fty  package  that  was  eating  up  a  vast  portion  of  the  FAA's  budget. 

er-increasing  pay  (it  rose  75%  between  1998  and  2005)  left  the  FAA  with  less 
pend  on  modernization  projects  that  might  actually  alleviate  some  of  the 
lig  "counterproductive,"  as  Mr.  Schumer  claimed,  Ms.  Blakev's  victory  over 
essary  step  toward  to  getting  the  air-travel  system  back  on  track. 

Jeally  wants  to  do  some  good  for  New  York's  airports,  though,  he  might  talk 
1  Connecticut.  The  delays  in  New  York  and  around  the  country  are  not 
y  have  a  tendency  to  become  tangled  in  the  competing  interests  of  NTmbyism, 
:ourse,  money.  When  you  combine  those  with  Washington  partisanship,  the 
Ihe  air. 


e  the  three  main  New  York-area  airports  that  Mr.  Schumer  is  so  concerned 
about.  ISHe  airspace  pround  New  York,  including  the  flight  paths  for  getting  in  and  out  of  the 
area,  dates  back  decades,  when  traffic  was  much  lighter.  The  FAA  has  been  trying  for  years  to 
"redesign"  it,  in  its  phrase,  to  allow  more  planes  more  ways  to  get  in  and  out.  But  activists  in 
Connecticut  don't  want  the  planes  going  over  their  houses,  and  neither,  it  seems,  does  anyone 
else.  So  redesign  becomes  apolitical  football,  and  the  planes  sit  on  the  taxiway. 

The  good  news  is  that  the  FAA  is  supposed  to  issue  a  redesign  rule  for  New  York  by  the  end  of 
August.  The  bad  news  is  that  even  if  the  new  routes  work  as  intended,  the  result  will  just  be 


4 


Pressure  of  Global  Warming  Concerns 
on  Flight  Efficiency  and  Fuel  Consumption' 
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Needless  delays  adcJJtMponution 

By  David  Learmount 

Eurocontrol  demands  more  efficiency  from  air  traffic  manag 
and  fewer  carbon  emissions 

Aircraft  flying  in  European  airspace  last  year  poured  thousands  oft 
warming  carbon  dioxide  into  the  sky  unnecessarily  just  because  of 
management  inefficiency,  according  to  the  Eurocontrol  Performanc< 
Commission  (PRC). 

The  report  on  calendar  year  2006,  published  on  11  May,  shows  tha 
delays  have  been  increasing  for  three  years  in  a  row,  and  PRC  chai 
Williams  says  that  the  need  to  improve  ATM  efficiency  is  rising  as  a 

This  excess  of  emissions  results  from  inefficiencies  in  the  continent 
which  mean  every  flight  travels  nearly  50km  (27nm)  farther  throuc 
needs  to  in  order  to  reach  its  destination,  the  PRC  reports.  The  Eur 
has  set  ATC  service  providers  the  target  of  eliminating  this  problem 
and  2010,  saving  an  estimated  2.3  million  tonnes  of  C02  emissions 
airline  costs  by  a  billion  euros. 
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Concern  grows  over  pollution  from  jets 

By  Gary  Stoller,  USA  TODAY 

Aviation  and  the  environment  are  on 
course.  The  number  of  airline  flights 
growing  and  expected  to  skyrocket  o' 
coming  decades.  Aircraft  emissions  [ 
and  threaten  by  2050  to  become  one 
largest  contributors  to  global  warminc 
scientists  have  concluded. 

Much  remains  unknown  about  climati 
and  the  role  aviation  plays,  though  cli 
scientists  express  particular  concern 
emissions  in  the  upper  atmosphere,  v 
warming  effect  from  some  pollutants 

Now,  aviation  is  believed  to  be  less  a 
Earth's  warming  than  power  plants  or 
traffic.  But  its  emissions  are  consider 
New  York-to-Denver  flight,  a  commer 
generate  840  to  1 ,660  pounds  of  cart 
per  passenger.  That's  about  what  an 
generates  in  a  month. 

With  the  projected  explosion  in  world\ 
air  pollution  from  aviation  is  a  growint 
among  scientists,  and  it's  drawing  inc 
scrutiny  from  governments,  particular 

"It's  an  issue  that  has  to  be  addresse 
Brenda  Ekwurzel,  a  climate  scientist 
of  Concerned  Scientists,  an  environnr 
advocacy  group. 

David  Travis,  a  climate  science  profe 
University  of  Wisconsin-Whitewater,  j 
emissions  "are  currently  one  of  the  fa 
growing  contributors  to  global  warmir 
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News 

Rep.  Markey  Wants  FAA  To 
Account  For  Global  Warming  In 
NextGen  Plan 

Aviation  Daily 
08/16/2007 

John  M.  Doyle 


A  House  committee  chairman  wants  the  FAA  to  include  global 
warming  in  its  Next  Generation  Air  Transportation  system 
(Mestfieo)  planning. 

Rep.  Edward  Markey  (D-Mass.)  wrote  FAA  Administrator  Marion 
Aug.  14  to  express  concerns  that  the  agency  wasn't  taking 
global  warming  into  account  as  it  planned  the  future  or  air  traffic 
control.  "American  aviation  has  made  the  word  a  smaller  place 
and  now  it  can  make  it  a  healthier  place  by  taking  action  on  global 
warming,"  said  Markey,  chairman  of  the  house  Select  Committee 
on  Energy  Independence  and  Global  Warming. 

The  letter  asks  FAA  to  report  back  by  Sept.  4  on  four  questions: 
What  does  NextGen  consider  aviation's  current  and  anticipated 
impact  on  global  warming?;  How  many  tons  of  carbon  dioxide 
does  aviation  emit  on  a  yearly  basis  in  the  U.S.  —  both  in  the  air 
and  on  the  ground?;  What  strategies  is  NextGen  considering  to 
address  emissions  at  airports?;  and  how  far  along  is  NextGen  in 
developing  a  national  roadmap  on  the  viability  of  alternative  fuels 
for  aircraft? 


Source  Scupps  Institution  of  Oceanography  and 
the  Pacfc  North  west  National  Laboratory’s  Global 
Change  Research  Institute  at  the  University  ot 
Maryland 


By  Ron  Coddington  and  Josh  Hatch.  USA  TODAY 


The  European  Union  is  considering  s 
on  aircraft  emissions,  an  action  strongly  uppuseu 


by  the  White  House  because  of  its  potential  errect 
on  U  S  airlines 


By  Andre  Penner,  AP  file 

A  Boeing  737  lands  in  Brazil.  Jet  emissions  such  as 
carbon  dioxide,  nitrogen  oxide  and  water  vapor  can 
contribute  to  climae  change. 


Carbon  dioxide  represents  about  80 
percent  of  the  man-made  greenhouse 
gases  blamed  for  global  warming  Though 
experts  have  called  for  the  United  Stales 
and  other  nations  to  cut  emissions  of 
C02.  current  trends  show  atmospheric 
C02  concentration  would  double  by  the 
end  of  Ihe  century. 
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Baseline  Values 


Why  NextGen? 


Shift  in  passengers  per  flight 
(e.g.,  A380,  reverse  RJ  trend, 
vhiqher  load  factor) _ 


2014  and  later  Baseline  analysis 
will  use  OEP  &  FAG"  Rapacities 


pp 


Passengers 


1.8-2.4X 


Biz  shift 

•  Smaller  aircraft,  more  airports 


Increase  of  over  10 
passengers  per  flight 


~3X 

Note:  Not  to  scale 


Flights 


Biz  shift 

•  2%  shift  to  micro  jets 


2014 


2025 


20?? 


2004  Baseline 

•  Current  Flight  Schedule 
^  •  Current  Capacities 


Growth  in  volume  and 
complexity  of  operations 
Scalable  to  encompass  a 
range  of  possible  futures 
Broader  diversity  of: 

-  Aircraft  performance 
characteristics 

-  Aircraft  capabilities 

-  Operator  business 
models 

Space  Operations 
Unmanned  Aerial  Systems 


Transformation  is  Needed  to  Accommodate 
Projected  Traffic  Levels  and  Characteristics 


MITRE 


What  is  NextGen? 


Transformation  goals: 

Leadership  in  global  aviation 
Scalable  up  to  3x  increase  in  capacity 

Ensure  our  national  defense  (readiness 
and  homeland  security) 

Enhance  the  environment  (noise,  air 
quality) 

Improve  safety 
Globally  harmonized 

Capabilities: 

Network-Enabled  Information  Access 
Performance  Based  Operations  &  Services 
Weather  Assimilated  into  Decision  Making 
Layered,  Adaptive  Security 
Position,  Navigation,  and  Timing  Services 
Aircraft  Trajectory  Based  Operations 
Equivalent  Visual  Operations 
Super  Density  Arrival/Departure  Operations 

MITRE 
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NextGen  Enterprise  System 
Engineering  Challenges 


Strategic 
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Stakeholder 
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Enterprise  System  Engineering  Profiler  ™ 


MITRE 


NextGen  Challenges:  Strategic  and 

System  Contexts 


•  System  Context 

-  System  Behavior 

•  NextGen  must  be  flexible  to 
meet  a  range  of  air 
transportation  system  futures 

-  Desired  Outcome 

•  Transformed  air  transportation 
system  that  leverages  new 
technologies  and  requires 
policy  and  roles  and 
responsibility  changes 


•  Strategic  Context 

-  Scope  of  Effort 

•  Interdependencies  of  all 
elements  contributing  to  air 
transportation 

-  Mission  Environment 

•  Mission  evolving  to 
accommodate  new  types  of 
operations 
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NextGen  Challenges:  Stakeholder  and 
Implementation  Contexts 


•  Stakeholder  Context 

-  Stakeholder  relationship 

•  Large  and  diverse  stakeholder 
community 

-  Stakeholder  Involvement 

•  Diversity  of  stakeholders  leads 
to  conflicting  objectives;  e.g., 

-  NextGen  funding  mechanisms 

-  Aircraft  equipage  mandates 

-  Airspace  access 


•  Implementation  Context 

-  Acquisition  Environment 

•  Synchronization  of  research, 
development,  and 
implementation  of  multiple 
government  agencies  and  the 
private  sector 

-  Scale  of  Effort 

•  Flexibility  required  to 
accommodate  multiple  user 
operating  models 
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Process  for  Achieving  NextGen 


MITRE 
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Baseline  and  Assess 
Today’s  Performance 


CAPSTONE  REQUIREMENTS  DOCUMENT 


GLOBAL  INFORMATION  GRID  (GIG) 


JROGM  134-01 
30  August  2001 


*  Understand  Federal  agency  and  private  sector  plans, 
including  architectures 

*  Baseline  current  capabilities  and  performance 
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Concept  of  Operations 


Joint  Planning  and 
Development  Office 


Concept  of  Operations 

for  the 

Next  Generation  Air 

Transportation 

System 

Version  2.0 
14  June  2007 


Next  Generation  Air  Transportation  System 


Describes  national  airspace 
system  (NAS)  in  2025 

-  Highlights  differences  from 
today’s  operations 

Presents  an  “aggressive”  set  of 
concepts 

-  Maximize  benefits  and 
flexibility  to  users 

Identifies  key  research  issues 
needing  resolution 
Highlights  areas  for  policy 
decisions 

Many  possible  futures 

-  Down-selection  and 
refinement  of  concepts  to 
occur  through  research  and 
policy  decisions 
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Operational  Improvements 


ID 

Task  Name 

Capacity  Management 

01-0345 

SUA  Airspace  Management  -  Level  1  Real-time  Scheduling  Information 

01-0310 

Improved  General  Aviation  Access  to  Traverse  Terminal  Areas 

01-0350 

Flexible  Routing 

01-0351 

Airspace  Reconfiguration  -  Level  1  Limited  Dynamic  En  Route 

01-0361 

Flexible  Resource  Allocation  for  Airspace  Management 

01-0307 

Airspace  Reconfiguration  -  Level  2  Limited  Dynamic  Arrival/Departure 

01-0337 

Row  Corridors  -  Level  1  Static 

01-0357 

Airspace  Reconfiguration  -  Level  3  Dynamic  En  Route 

OI-0365 

SUA  Airspace  Management  -  Level  2  Improved  Coordination 

01-0355 

Dynamic  Airspace  Reclassification 

01-0342 

Airspace  Reconfiguration  -  Level  4  Dynamic  Arrival/Departure 

OI-036S 

Row  Corridors  -  Level  2  Dynamic 

P7  I  OS  I  09  I  10  111  I  12  I  1o  I  U  |r  15  I  16  I  IT  I  13  I  19  I  20  I  21  I  22  I  I  2i  la  I  26~| 
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•  Definition  -  A  change  in  operations  that  produces  a  beneficial 
result  and  moves  the  air  transportation  system  toward  the  2025 
NextGen  Goals  and  Objectives 

•  -  130  Ols  are  grouped  to  describe  the  operational  transition  path 
toward  the  future 

•  Ol  roadmaps  span  the  strategies  and  key  capabilities  described  in 
the  Integrated  Plan 
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Enterprise  Architecture 


Joint  Planning  and 
Development  Office 

enterprise  Architecture  V2.Q 

for  the 

The  Next  Generation  Air 
Transportation  System 


22  June  2007 


•  Federal  Enterprise  Architecture 
Framework  (FEAF)  and 
extensions  for  air 
transportation 

•  Department  of  Defense 
Architecture  Framework 
(DODAF)  -  selected  and 
tailored  operational,  system, 
and  technology  views 

•  Federal  Transition  Framework 
(FTF)  -  Air  Transportation  Line 
of  Business  proposal 


•  Tool  to  relate  and  integrate  NextGen  Federal  agency  and 
private  sector  efforts 

-  Planning,  portfolio  management,  and  system  acquisition 

-  Key  purpose:  support  OMB  investment  decision  process 

•  Models  to  describe  the  NAS  from  operational,  information, 
systems,  technology,  and  performance  perspectives 
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Determine  Program  and 
Research  Needs 


Joint  Planning  and 
Development  Office 

Research  and 
Development  Plan 

for  the 

Next  Generation  Air 
Transportation 
System 

FY  2009  -  FY  2013 

31  August  2007 

NextGen 

Next  Generation  Air  Tramspoe-tatEoft  System 

I 


JPDO  working  with  government 
agencies  and  private  sector  to 
determine  program  and  research 
needs 

-  Considers  agency  specific 
strategic  plans 

CONOPS  has  top-level  research 
issues 

FY09  to  FY13  Research  and 
Development  Plan  developed  to 
support  budget  formulation 
Demonstrations  identified  to 
assess  operational  concepts, 
system  implementations,  or 
technologies 
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Define  and  Implement 
Incremental  Solutions 


Full  range  of  DOTMLPF  solutions  applies  across 

public  and  private  sectors 

Policy  issues  addressed  in  an  integrated  manner 

-  Examples:  surveillance  integration,  navigation 
system  backup,  and  equipage  for  required 
performance 

Portfolio  management 

-  Investments  selection  (program  and  research) 

-  Collaborative  effort  among  stakeholders, 
including  the  agencies,  and  OMB 

-  Business  case  includes  benefits  and  costs 

-  “Portfolio”  view  is  needed  to  justify 
investments  contributing  to  Ols  -  not  a  program 
by  program  view 

NextGen  Integrated  Work  Plan 


Joint  Planning  and 
Development  Office 


Integrated  Work  Plan 

for  the 

Next  Generation  Air 

Transportation 

System 

Version  QJ 
31  July  5007 


Trj  nLpD  rl  jtlan 
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-  Time  phased  plan  consolidating  01  roadmaps,  research  plans, 
policy  needs,  and  other  implementation  supporting  material 
Agencies  and  private  sector  implement;  JPDO  monitors  and 
assess  progress,  including  performance  and  cost 


MITRE 


JPDO  Organization 


Governance  Advisors  Inter-Agency  Coordination 


*  Senior  Policy 
Committee 

•  Board  of  Directors 


•  REDAC  Executive 
Committee 

•  Institute 
Management 
Council 


•  Joint  Architecture  & 
Engineering 
Board 


Government  leadership  reflects  participation  from  NextGen  agencies 
Organization  tailored  to  achieve  enterprise  transformation 
JPDO  includes  government,  contractor,  and  Federally  Funded 
Research  and  Development  Center  (FFRDC)  managers  and  staff 
Working  Groups  responsible  for  domain-specific  products 
-  Include  government,  NextGen  Institute,  and  FFRDC  members 
Multilevel  governance  includes  cabinet  level  leadership  (SPC) 


MITRE 
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NextGen  Stakeholder  Participation 


Stakeholder 

Roles  and  Responsibilities 

Agencies 

Develop  and  review  overarching  products,  conduct 
government  research,  and  implement  government 
programs 

Private  sector 

Develop  and  review  overarching  products,  conduct 
private  sector  research,  and  implement  private  sector 
programs 

Office  of 
Management  and 
Budget 

Review  NextGen  EA  and  review  of  NextGen  Business 

Case 

Congress  and 
Government 
Accountability 
Office 

Conduct  review  of  NextGen  progress  reports  and 
selected  products,  and  review  of  JPDO  effectiveness 

Department 

Inspectors 

General 

Conduct  review  of  JPDO  effectiveness 
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Lessons  Learned 


•  Governance  must  be  established  early  to  ensure  the  roles  and 
responsibilities  of  participating  government  organizations  and 
industry  stakeholders  are  clearly  defined  and  described 

-  Boundaries  and  activities  which  delineate  the  “who  does  what” 

-  Scope  and  depth  of  the  interoperability  required  between  multiple 
agencies’  and  industry,  various  activities  and  systems/applications 

-  Information  exchange  required  among  participants 

•  JPDO  could  not  reuse  existing  single  agency  processes  and 
products  without  changes  to  plan  and  oversee  implementation  of 
NextGen 

-  Single  agency  coordination  processes  and  products  were  not 
sufficient  to  address  mission  needs,  although  most  came  from  the 
participating  agencies  themselves 

-  Need  to  consider  public-private  partnership,  multi-agency  operations, 
cross  agency  investments,  and  long  term  planning  horizon  influence 
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Lessons  Learned 


•  Multi-agency  organizations  require  a  common  way  to  describe  key 
aspects  of  cross  agency  planning 

-  Common  products  and  process  flow 

-  Level  of  detail  varies  by  area  -  overlap  with  agency  products  expected 

-  Concerns  differs  across  stakeholders  -  White  House,  Congress,  industry 
groups,  etc. 

-  Visibility  into  evolving  products  needed  while  protecting  sensitive 
information 

•  JPDO  continues  to  deal  with  complexity  that  increases  with  scope, 
diversity  of  stakeholders,  time  horizon,  applicable  technologies, 
policy  areas,  etc.  -  more  complex  than  single  agency  effort 

-  Processes  continue  to  evolve  to  achieve  the  products  that  are 
understandable  by  the  various  agencies  and  industry 

-  Long  time  horizon  means  that  “design  space”  will  not  close  until 
R&D  complete 

•  Public-private  partnerships  require  government  and  industry 
leadership  and  staffing 

-  Work  groups  have  government  and  private  sector  co-leads  and 
participation 

-  Industry  representatives  are  members  of  the  Integration  Council  MITRE 


21 


— 


Summary 


*  NextGen  is  a  pathfinder  for  addressing  large  scale, 
multi-agency  enterprise  system  engineering 
challenges 

-  Joint  and  sometimes  conflicting  missions 

-  Transformation  of  services 

-  Reducing  costs  for  government,  industry  and  the  public 

*  Multi-agency  and  public-private  efforts  involve  higher 
complexity  and  additional  products  and  processes 
compared  to  single  agency  programs 

*  Progress  depends  on  satisfying  multiple  stakeholders 
while  maintaining  focus  on  most  important  products 
and  impacts 


This  is  the  copyright  work  of  the  MITRE  Corporation  and  was  produced  for  the  US  Government  under  Contract  Number  DTDA01-01-C-00001  and  is  subject  to  Federal  Aviation  Administration  Acquisition 
Management  System  Clause  3.5-13,  Rights  in  Data  General,  Alt.  Ill  and  Alt.  IV  (Oct.,  1996).  *  m  .yn|- 

MITRE 
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For  More  Information 


*  Joint  Planning  and  Development  Office: 

-  http://www.ipdo.aov 

•  NGATS  Institute: 

-  http://www.ncat.com/nqats/index.html 
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National  Defense  Industrial  Association  Carole.LeBlanc@osd.mil 

10th  Annual  Systems  Engineering  Conference  703.604.1934 

October  25,  2007 
San  Diego,  California 

DoD’s  Emerging  Contaminants 

Program 

Carole  LeBlanc,  Ph.D. 

Special  Expert,  Emerging  Contaminants 
Office  of  the  Deputy  Under  Secretary  of  Defense 
(Installations  &  Environment) 
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•  ■  *  ■  *  ■ 


Emerging  Contaminants  Directorate 


Presentation  Overview 

The  Department  of  Defense’s  (DoD’s) 
Interpretation  of  Emerging  Contaminants 

Opportunities  for  Improving  Risk  Management 
(RM)  Options  at  DoD 

♦  Processes  for  Assessing  and  Managing  Risks 

♦  ‘Watch’  and  ‘Action’  Chemicals 

EC  Program  Highlights 

♦  Proactive  Approach:  What  Program  Managers  (PMs)  and 
Others  Can  Do 

»  Materials  of  Evolving  Regulatory  Interest  Team  (MERIT) 

»  Websites 


Emerging  Contaminants  Directorate 


National  &  International  Interest  in  ECs 


Press 

♦  ES&T  Magazine  -  Dec  06 

♦  National  Geographic  Magazine  -  Oct  06 

Conferences/Hearings/Reports 

♦  AWRA  Spring  Conference  -  June  07 

♦  USGS  Survey  of  139  streams  in  30  states 

»  ECs  found  in  80%  of  streams 

♦  Renewable  Nat’l  Resources  Foundation 
Report 

Regulations/Mandates 

♦  New  Executive  Order  (January  24,  2007) 

♦  European  Union  -  REACH  (June  1, 2007) 


What  Is  an  Emerging  Contaminant? 

At  the  Department: 

Emerging  Contaminants  (ECs)  are  defined  as 

*:  Chemicals  &  materials  with 

♦  Perceived  or  real  threat  to  human  health  or  environment 

♦  Either  no  peer  reviewed  health  standard  or  an  evolving 
standard 

*:  May  have  • 


♦  Insufficient  human  health  data/science 

♦  New  detection  limits 

♦  New  exposure  pathways 


■  ■  *  ■  *  ■  *  ■ 


Emerging  Contaminants  Directorate 


How  Can  ECs  Affect  DoD? 

Adverse  health  effects  on  operating  forces, 
DoD  employees,  and/or  public 

♦  Human  health  protection  paramount 

Reduced  training/readiness 

♦  Restrictions  on  use  of  ranges 

Restricted  or  non-availability  of  material 

♦  Adverse  impact  on  mission-critical  applications  & 
industrial  base 

Increased  O&M  and/or  cleanup  costs 

♦  Resource  drain  from  mission  needs 


•  ■  *  ■  •  ■  *  ■ 


Emerging  Contaminants  Directorate 


The  Vision 

Imagine  if  the  largest  industrial  complex  in  the  nation  could... 

Predict  which  chemicals  it  uses,  or  might  use, 
pose  human  health  and  environmental  risks 
and  have  an  evolving  regulatory  status 

Develop  a  consensus  evaluation  of  types  & 
magnitudes  of  the  risks  in  using/releasing 
such  chemicals 

Develop  risk  management  options  and  invest 
in  high-payback  actions 

Achieve  and  measure  risk  reduction 


Emerging  Contaminants  Directorate 


Strategic  Priorities  to  Achieve  the  Vision 


Strategic 
Process 
Improvements 


National  Level 
Issues 


Engage  Internal 
&  External 
Stakeholders 


Identify,  Assess 
&  Manage  DoD 
Risks 


DoD,  Federal,  State, 
NGOs,  &  Industry 


Internal  DoD  & 
Industry  Partners 


Emerging  Contaminants  Directorate 


EC  “Scan-Watch-Action”  Process 


Over-the-horizon 


scan 


I 


watch 


it 


action 


EC  News 


Possible  OoD  impacts 


Phase  I 
Assessment 


Probable  high  DoD  impacts 


Phase  I! 
Assessment 


Review  literature,  periodicals, 
regulatory  communications, 
etc.  to  Provide  early  warning  of 
ECs  of  possible  interest  to  DoD 

Monitor  events;  Conduct  Phase 
I  qualitative  impact  assessment 
to  assess  impacts  to  DoD 

Conduct  Phase  II  quantitative 
impact  assessment  with  risk 
management  options  to  create 
strategic  investment  options  for 
enterprise  consideration 


risk 


RM  Options  to  Governance  Council 


Emerging  Contaminants  Directorate 


Phase  I  EC  Impact  Assessment 


1. 

2. 


Probability  of  Regulation/Re-regulation 


Impact  on  DoD  Functional  Categories 


Environment 
Safety  & 
Health 

Readiness  & 
Training 

Acquisition 

Production, 
O&M  and 
Disposal 

Cleanup 

m 

CD 

® 

® 

® 

® 

® 

® 

® 

® 

CD 

CD 

CD 

CD 

CD 

► 


3.  Results 


*  Decision  -  Move  to  Action  List? 

*  Initial  Risk  Management  Options 


Emerging  Contaminants  Directorate 


Plotting  EC  Risk  to  DoD 


High  risk  at  top  right 

Risk  management 
actions  move  ECs 
to  lower  left... 
lower  risk 

Seek  to  quantify 
risk  reduction 


•  -  High 
©  -  Med 

O  -  Low 


\  Risk  Change 


Emerging  Contaminants  Directorate 


Integrated  Risk  Management  (RM) 


a> 


LO 


e 

© 

e 

0 

©  r 
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Accept  Risk 


RM  Options 

•  Fill  tox  science  gaps 
• RDT&E 

•  Material  substitution 

•  Process  changes 

•  Regulatory  engagement 

•  Stockpile  material 

•  Exposure  assessment  & 
monitoring 

•  Personal  Protective 
Equipment  (PPE) 

•  Acquisition  changes 

•  Benchmark  with  industry 

•  Risk  communication 

•  Training 


Emerging  Contaminants  Directorate 


Example:  TCE  and  the  Relative  Risks  to 

EH&S  and  Cleanup 


Likelihood  of  Adverse  Impact 

■ 

* 

m 

A 

i 

m 

L  Severity  of  Impact  H 

TCE  Phase  1  Impact  Assessment 

Completed  October  2006 

Legend 

♦  ES&H 

■  Readiness  &  Training 
a  Acquisition/RDT&E 

*  O&M  of  Assets 

t  Cleanup _ 


Emerging  Contaminants  Directorate 


Example:  Beryllium  and  the  Relative  Risks  to 
EH&S  and  Readiness  &  Training 


H 


o 

<0 

Q. 

£ 

<D 

S2 

o> 

> 

■O 

< 


■a 

o 

o 


Q) 


u 

( 

1 

■ 

t 

▲ 

■ 

Severity  of  Impact 


H 


Beryllium  Phase  1  Impact  Assessment 

Completed  March  2007 


Legend 

♦  ES&H 

■  Readiness  &  Training 
a  Acquisition/RDT&E 

*  O&M  of  Assets 

t  Cleanup _ 


Emerging  Contaminants  Directorate 


Chemical  and  Material  Differences 


Watch  List 

<  May  impact  DoD 

•:  Limited  analysis  of  impact  - 
more  qualitative 

•:  Monitor  external  actions 
*:  Updated  regularly 
*:  Short  info  sheets  developed 
*:  Minimal  resources  expended 


Action  List 

•:  Likely  to  impact  DoD 

*:  Detailed  analysis  of  impact  - 
more  quantitative 

*:  Take  RM  actions 

*:  Executive  info  sheets 
developed 

*:  Significant  resources  may  be 
expended 

•:  “Material  champion”  assigned 


Emerging  Contaminants  Directorate 


Chemicals  and  Materials  Identified  Thus  Far 


PFOAs  Beryfliu 


•  • 


Emerging  Contaminants  Directorate 


EC  Program  Summary 

EC  management  required  new  thinking 

♦  Make  targeted  investments  before  regulatory  action 

♦  Base  decisions  on  life  cycle  costs 

♦  Craft  process  for  tracking,  assessing  and  developing 
risk  management  options 

»  Leverage  existing  assets/resources 

EC  management  required  new  organization 

♦  Created  Emerging  Contaminants  Directorate 

♦  Established  funding  line  within  DoD  environmental 
budget 

♦  Launched  executive  level  EC  Governance  Council  to 
enhance  corporate  decision-making 


•  ■  *  ■ 


Emerging  Contaminants  Directorate 


EC  Program  Highlights 

Keep  Program  Executive  Offices  (PEOs)  and 
Program  Managers  (PMs)  informed  on 

♦  New  ECs 

♦  Risks 

♦  Risk  management  options 

Some  EC  Products 

♦  Newsletter 

»  Provides  brief  news  events  &  links 

♦  Scan-Watch-Action  List  and  Impact  Assessments 

♦  Web  Site 

»  EC  database  and  information 


Emerging  Contaminants  Directorate 


Become  Involved  in  the  EC  Process 


Including  MERIT,  a  virtual  team 


Emerging  Contaminants  Directorate 


DoD  EC  Web  Sites 


MERIT  -  Microsoft  Internet  Explorer  provided  by  Comcast 


^iQjx. 


File  Edit  View  Favorites  Tools  Help 


^  Back  t  =>  T  @  ^Search  Favorites 


a 


Address  |  https://www.denix.osd.mil/denix/Public/Librarv/MERIT/merit.html 


~*~1 


Google 


~^|  Go  |  ^  §1 


l  780  blocked 


%  AutoLink  ▼  ,_j  AutoFill 


;ii  » 


Settings  t  ^  t 


Emerging  Contaminants  Directorate 


Emerging*  Contaminants  (EC) 


DoD  Efforts 


News  Room 


Resources 


Calender 


Quick  Links 

*  Policy  &  Guidance 

•  Glossary 


DoD  Site:  http://intranet.dodmeritinfo.net 


Contact  Us 


Feedback 

We  value  your  input. 
Please  let  us  know  how  we 
can  serve  you  better... 


DoD  is  committed  to 
protecting  the  public,  our 
work  force,  the 
environment,  and  our 
national  security. 


What  is  an  Emerging 
Contaminant? 


What's  New 


Announcements: 


Emerging  Contaminants  are 
chemicals  or  materials  of 
interest  that  are  characterized 
by: 


DoD  Account  Login: 


About  MERIT 

The  Materials  of  Emerging 
Regulatory  Interest  Team 


July  27,  2006 

MAS  releases  its  report  "Assessing 

Human  Health  Risks  of  TCE:  Key 

Scientific  Issues" 


A  perceived  or  real  threat  to 
human  health  or 


January  26,  2606 

ERA  Issues  Guidance  for  Protective 


Emerging  Contaminants  Directorate 


The  Take  Home  Message 

Proactive  vs  Reactive 

Readiness  Impacts 

Cleanup  Costs 

Compliance  Costs 

Health  Claims 

Platform/Facilities 
Life  Cycle  Costs 

$$$  $$$ 

Small  investment  here  Large  impact  here 

Potential  Large  Payback 
Protects  Mission,  People  and  Assets 


Emerging  Contaminants  Directorate 


Otherwise... 


The  real  reason  dinosaurs  became  extinct. 


USAF  Type  Certification  of 
Commercial  Derivative  Aircraft 


25  October  2007 


Joel  Ligon 

Program  Manager/Aviation  Safety  Engineer 
Federal  Aviation  Administration 
Military  Certification  Office 
ACE-1 00M 
8200  East  34th  Street  North 
Wichita,  KS  67226 
joel.ligon@faa.gov 
Phone  (316)  350-1593 


Thomas  Morgan 

Technical  Specialist  for  Airworthiness  of  Commercial 
Derivative  Aircraft 
Airworthiness  Group 
ASC/ENSI 

2530  Loop  Rd  W.  B-560 
WPAFB,  OH  45433-7101 
Thomas.morgan@wpafb.af.mil 
Phone  (937)  255-1814;  DSN  785-1814 
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DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited  (ASC/PAM 
Document  Number  ASC  07-0315  dated  25  Sep  2007). 


Course  Outline 


•  Airworthiness  Authority  -  FAA  and  USAF 

•  Air  Force  Airworthiness  Tools  -  TACCs  and  MACCs 

•  When  Civil  and  Military  processes  collide 

•  Managing  the  seam,  maintaining  FAA  pedigree  and 
safety  levels  on  military  installations 

•  New  FAA  Order  81 10.101 

•  Levels  of  Certitude  -  Civil  and  Military 

•  Issues  when  mixing  fish  and  foul 

•  Summary 

•  Conclusions 
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Airworthiness  Authority 


Title  49 

Department  of 


Transportation 


United 
States 
Code 


40113  FAA  Administrator: 

(a)  Has  discretionary  authority 

(b)  May  indemnify  others  to  act... 
44701  (a)  in  promoting  safety  may 

(1)  Establish  minimum  standards 

(2)  Issue  regulations 


Code  of  Federal  Regulations 

lTitle  14 
Aeronautics 
and  Space 


'H  a 

Aeronautics 

and  Space 

.35  ■& 

TO 

Title  14 

Parts  1  to  199 

Revised  January 

1, 1998 

1 

Title  10 

Department  of 
Defense 


8013  Secretary  of  the  Air  Force 

(b) (3)  Supplying,  (3)  Equipping, 
(5)  Training,  (9)  Administering 

(including  welfare) 

(c) (2)  Formulation  of  policies 


0/  A 

<b  &  ta  MiUV-UIP 


<0 


<0  C. 


< o 


™  _ ^  _ 


AFPD  62-6  USAF  Aircraft  M 
Airworthiness  Certification 


~AIR  FORCE  POLICY BIRIC TUT  62-S 
1  OCTOBER  2W8 

DcvelopmeiOid  Engineering 

USAF  .AIR  CR.4FT  .AIR  WDM  JBBffiSS 
CERTIFICATION 


NOTIC  E  :  ltd s  publication  ls  available  digitalLy  on  [be  AFDPO  WU.TV  site  at:  http .  afpubi.hq. af  mil. 


OPR:  SAF  AQRE  (Mr.  Paul  Hfoscb) 


Certified  by:  SAF,-  AQR  iIJt.  Donald  C.  Daniel) 
Page;-:  5 
Distribution:  F  . 


AJ rc rvr  ml  r~  r~  ii  '  l .  Hu  '■  i  I  m  ■  I  ill  1 1  i  I  ■  i  1  'i1  i  ii  hi  ~~ r  ~iil  ■mr  rUf  ivri^i-/. f 

nvaxcrafi  art.  [bus  the  Air  Force  is  die  responsible  agent  for  verification  of  airwontixEss  TbLspbSs 
vtgy^labliaaeE  the  requirement  for  airworthiness  certification  of  USAF  aircraft  by  the  responsible  sljh 
manager  ana  estaciiillUa  ike  PJrwoTthxsg  s  Cartiftsanpu  Criirrii.'  Coun-l  D-.ni  I.AL~bi.i.  Ini-;  polio1 
applies  to  all  US  Air  Forte  .aircraft,  including  these  of  the  Air  National  Guard  (ANG)  and.  US  Air  Force 
Reserve  C  Dimnand  (AFR.C). 


r^AiTwarthines-E-  certificatiDn  is  required  for  aLL  USAF  aircraft  entering  or  currently  u 
Airawthriness  verification  e Ira  11  signify  compliance  to  the  Airworthiness  Cwiification  Criteria^ 
es-tablished  by  the  AC^.  Tbe  single  manager  fSM)  for  the  aircraft  is  the  airworthiness  certificatioD  offi¬ 
cial.  Related  policy  le  contained  in  AFPD  62-4.  Standards,  of  AirwonhiiresE  for  Passenger  Carrying  Com- 
lercial  Derivative  Transport  Aircraft,  and  AFPD  62-5?  Standards  Of  Airworthiness  For  Commercial 
Dabvatiye  Hybrid  Aircraft. 


1.  Respon.EibLbd.es  and  A 

2.1.  SAF  AQ  will  ensure  acquisition  directives  and  polities  require  the  airwnnbiness  certification 

2.2.  USAFi'XO  will  ensure  airworthiness  certifLcanon  requirements  are  included  in  joint  and  Air 
Force  program  requirement  documents. 

2J.  USAF'IL  will: 

2J.1  Ensure  mamtenante  management  directives  and  pohties  {AFI 21-101.  Mamtenante  Man¬ 
agement  of  Aircrafti  AFI  21-102.  Depot  Maintenance  Management.  and.  AFI 21-107.  Maintaining 
CDinmercial  Derivative  Aircraft)  include  procedures  that  preserve  aircraft  airworthiness,  regard¬ 
less  of  maintenance  Location  (field  or  public'private  depots). 


“By  order  of  the  Secretary  of  the  Air  Force” 

“This  policy  establishes  the  requirement  for 
Airworthiness  Certification  of  USAF  Aircraft 
by  the  responsible  single  manager” 


“1.  General:  Airworthiness  Certification 
...applies  to  all  USAF  aircraft... Related 
Policy..AFPD  62-4,  AFPD  62-5..” 

“Section  2.8  Aircraft  Single  Managers  will: 

2.8.8  Prior  to  first  flight,  make  and 
document  a  positive  determination  of 
safety  of  flight  in  accordance  with  ASC 
guidelines. 
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AFPD  62-5  -  Hybrids 


liY  ORDER  OF  THE  ^V_ 

.  SEC  RETARY  OF  THE  AIR  FORCEJ 


- AIK  hO RLE  POLICY  DIREC  TIVE  62-5 

/  DEC  EMBER  199H 
De  ee  lap  mental  Engineering 

STANDARDS  OF  AIRWORTHINESS  FOR 
COMMERCIAL  DERIVATIVE  HYBRID 
AIRCRAFT 


NOTICE:  I  bis  publication  is  available  digitally  on  the  SAF/AAD  WWW  site  at:  hup://afpubs.hq.af.mij/ 
If  you  lack  access,  contact  your  Publishing  Distribution  Office  (MX)). 


OPR:  HQ  USAF/XOO-CA 
(I  .t  Col  Pamela  Hodge) 


Certified  by:  HQUCM  /XOO 
(Maj  Gen  Charles Jrl Henderson) 
Pages:  4 
Distribution:  F 


ThKjdtrcfuvc  establishes  policies  to  ensure  the  Air  Force's  commercial  derivative  hybrid  aircrafTtrsutlfiir 
perations,  surveillance,  training,  and  test  and  evaluation  maintain  high  levels  of  safety  and  to  ensure  tmN 
r  Force  docs  not  duplicate  activities  performed  by  the  Federal  Aviation  Administration  (FAA),  such  as 
Type  Certification  (TC)  or  Supplemental  Type  Certification  (S  IX’)  pertaining  to  those  aircraft.  The  intent 
s  Policy  Directive  is  to  cnsiiiv  the  highest  levels  of  safely  for  all  commercial  derivative  hybrid  J 


Force's 


I.  To  meet  military  requirements,  the  Air  t  oree  win  seek  to  procure  and  sustain  commercial  derivative 
hybrid  fixed  and  rotary  wing  aircraft  even  when  the  intended  use  of  such  aircraft  is  not  consistent  with 
original  design  or  adyil-Hui  will'll!  operation  d«>cs  not  c  x  i  .1.  I  In  ■  udui^smuch  of  the  life  cycle  costs 
associalcd^idrtlevdoping,  producing,  operating,  and  maintaining  govenimciTpflC'Hd^d  aircraft. 

To  gain  the  greatest  life  cycle  cost  savings,  the  Air  Force  w  ill  seek  to  ensure  its  commercial  derivative 
hybrid  aircraft,  to  the  extent  practicable,  comply  with  civil  airworthiness  standards  set  by  the  Federal  > 
ation  Regulations  (FAR)  Commercial  aircraft  arc  generally  required  to  comply  with  FAR  requirement sN 
and  Public  law  designates  the  F'AA  as  the  regulator  of  the  US  national  airspace  system  and  enforcer  of 
FAR  requirements.  However,  aircraft  owned  and  operated  by  the  Air  Force  are  public  aircraft  and  the  Air 
Force  is  the  responsible  agent  for  certification  of  airworthiness.  At  a  minimum,  the  Air  Force  will  use  the 
FARs  to  baseline  airworthiness  wherever  practicable.  The  System  Program  Director  (SPD),  who  has 
overall  responsibility  for  airworthiness,  will  use  Air  Force  modification  procedures  when  the  FARs  are^ 
inappropriate.  The  Program  Manager  is  responsible  for  the  engineering  design,  testing  and  modificatk) 
sjnade  to  commercial  derivative  hybrid  aircraft. 

5,  TheATH^uo^shall  seek  to  ensure  avionics  or  other  equipment  developed fopwetfifAir  Force  com¬ 
mercial  dmrrilhrTiTtTrhl  uirrriifl  mid  or  exceed  civil  design  si  yi|t  iivi^wr--irrr^-.M.  .-  with  the  airworthi¬ 
ness  and  operating  F  ARs  that  apply  to  the  aircraft  being  procured.  In  addition  to  the  FAR  requirements  or 
if  the  item  is  military  off-the-shelf,  military  design  standards  will  he  met  when  the  mission  dictates. 

4.  The  Air  Force  may  accept  and  use  F'AA  evaluations  and  inspections  to  reduce  duplicative  activities. 


“By  order  of  the  Secretary  of  the  Air  Force” 


“This  directive  establishes  policies  to  ensure  the  Air 
Force’s  commercial  derivative  hybrid  aircraft  ...maintain 
high  levels  of  safety  ...  and  to  ensure  the  Air  Force  does 
not  duplicate  activities  performed  by  the  ...FAA.  ...(to) 
ensure  highest  level  of  safety  for  all  commercial 
derivative  hybrid  Air  Force  aircraft.” 


2.  To  gain  the  greatest  life  cycle  cost  savings,  the  Air 
Force  will  seek  to  ensure  its  commercial  derivative 
hybrid  aircraft,  to  the  maximum  extent  practicable, 
comply  with  civil  airworthiness  standards  set  by  the 
Federal  Aviation  Regulations  (FAR).  ...  At  a  minimum, 
the  Air  Force  will  use  the  FARs  to  baseline 
airworthiness  whenever  practicable.  ...” 


4.  The  Air  Force  may  accept  and  use  FAA  evaluations 
and  inspections  to  reduce  duplicative  activities 
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AFPD  62-4  Passenger  CDA  # 


“By  order  of  the  Secretary  of  the  Air  Force” 


jRarsa 

f  I 


AIR  FORCE  POLICY  DIRECTIVE  62-1 
I  DECEMBER  I99R 
De  ve  lap  mental  Engineering 

STANDARDS  OF  AIRWORTHINESS  FOR 
PASSENGER  CARRYING  COMMERCIAL 
DERIVATIVE  TRANSPORT  AIRCRAFT  . 


NOTICE:  I  his  publication  is  available  digitally  on  the  SAF/AAI  >  WWW  site  at:  http://afpubs.hq.atymil 
If  you  lack  access,  contact  your  Publishing  Distribution  Office  (PDO). 


OPR:  HQ  USAI  /XOO-CA 
(I  t  Col  Pamela  Hodge) 

Supersedes  AFPD  62-4,  14  September  1993. 


Certified  by:  HQ  IAAF7XOO 
(Maj  (Jen  Charles  Hr  Henderson  > 
Pages:  4 
'  Distribution:  I 


mrective  establishes  policies  to  ensure  the  Air  Force's  passenger  carrying  commercial  denvath^ 
transport  aircraft  maintain  high  levels  <>i  safety  and  to  ensure  the  i\k  Force  does  noi  duplicate  activities 
itmed  by  the  federal  Aviation  Administration  (FAA),  such  as  Type  Certification  (TO  or  Supplement 
tal  TyprTTiiili  tin  ii  VFH  [vrnininy  in  those  aircraft.  The  imen|  nf  [hi-  P-  1 1  1‘ir'li  "TT  ensure 
the  highest  levels  of  safety  for  all  passenger  carrying  commercial  transport  Air  Force  aircraft. 


SUMMARY  OF  REVISIONS 

This  document  is  substantially  revised  and  must  be  completely  reviewed. 

This  revision  changes  the  title  from  Civil  Airworthiness  Standards  for  Transport  Aircraft  to  address  only 
passenger  carrying  commercial  derivative  transport  aircraft;  establishes  to  the  extent  practicable  a  single 
level  of  safety  for  aircraft  with  the  passenger  carrying  mission;  and  changes  the  metric  for  measuring 
compliance  to  reduce  redundant  record  keeping  (paragraph  8).  Transport  aircraft  which  accomplish  mis¬ 
sions  other  than  passenger  carrying  are  addressed  in  AFPD  62-5. 

I.  The  Air  Force  is  able  to  perform  many  of  its  passenger  carrying  missions  with  commercial  derivative 
transport  aircraft.  Procuring  such  aircraft  saves  costs  associates!  with  developing,  producing,  operating, 
and  maintaining  entirely  new  aircraft  throughout  the  life  cycle  of  the  program.  Therefore,  when  the 
intended  use  of  such  aircraft  will  be  limited  to  applications  comparable  to  civil  passenger  operations,  the 
Air  Force  will  seek  to  procure  and  sustain  commercial  derivative  fixed  and  rotary  wing  aircraft.  Public 
law  designates  the  Federal  Aviation  Administration  (FAA)  as  the  regulator  of  the  US  national  airspace 
system.  Commercial  aircraft  are  generally  required  to  comply  with  Federal  Aviation  Regulation  (FAR) 
requirements.  This  directive  establishes  policies  that,  subject  to  the  waiver  provisions  below,  seek  to 
ensure  that  the  Air  Force’s  commercial  derivative  transport  aircraft  used  for  carrying  passengers  comply 
with  applicable  FAR  requirements,  and  that  the  Air  Force  does  not  duplicate  activities  performed  by  the 
FAA  such  as  the  issuance  of  Type  Certification  (TC)  or  Supplemental  Type  Certification  (STC)  pertain¬ 
ing  to  those  aircraft. 
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...To  maintain  high  levels  of  safety  and  to 
ensure  the  Air  Force  does  not  duplicate 

FAA  Activities 

◄ — 

2.  ..meet  or  exceed  civil  design  standards  in 
accordance  with  airworthiness  and 
operating  FARs..  ;military  equipment  may 
be  used  but  must  be  reflected  as  deviations 
to  type  design... 

◄ — 

3.  ..avionics  must  meet  or  exceed  FAR 
rqts...seek  to  obtain  Equiv  Level  of  Safety 
related  to  equipment  carried  on  civil 
missions 

◄ — 

6.3  ...  MAJCOM  maintain  to  FAA 

Standards;  6.5  May  request  waiver  after  “all 
possible  solutions  to  resolving  FAR  issues 
have  been  exhausted.”  6 

AFI  21-107  Maintenance 


Airworthiness  Tools 


•  MIL-HDBKS 

-  514  Operational  Safety  Suitability  and  Effectiveness  for  the 
Aeronautical  Enterprise 

-  516  Airworthiness  Certification  Criteria 

•  Airworthiness  Certification  Circulars 

-  #4  Certification  Basis 

-  #x  Modified  Certification  basis 

•  Policy  Memorandums 

-  AFMC/EN  28  Jan  2002,  Review  of  TACC 

-  ASC/CC  19  Jul  2001  Notification  of  Airworthiness  Certification 

•  Each  program  generates  its  own  Taylored  Airworthiness 
Criteria  Checklist  (TACC)  for  the  baseline  and  Modified 
Airworthiness  Criteria  Checklist  (MACC)  for  follow  on 
mods. 
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MIL-HDBK-514 


Section  11.2.1  “Judged  to  be  airworthy  if... 
meets  approved  set  of  criteria  ...  defined  in  MIL- 
HDBK-516” 

Section  11.10.1  “Provide  written  notice  to 
ASC/EN  on  certification”,  gives  guidance  on 
reportable  modifications 

Section  11.10.2“...  accomplished  by  developing 
and  maintaining  ASC/EN  coordination  on  a 
(MACC)  for  each  reportable  modification” 


MIL-HDBK-516 
TACC  and  MACC 


•  Tailored  Airworthiness  Certification  Criteria  (TACC)  - 

Use  for  New  Air  Force  Aircraft  -  Certification  basis 

-  Documents  airworthiness  criteria,  requirements  and  methods  of 
compliance  used  in  development  of  the  aircraft 

-  Provides  a  baseline  for  configuration  control  throughout  the  life 
of  a  system  (ACC#4,  pgl) 

•  Modified  Aircraft  Certification  Criteria  (MACC)  -  Use 

when  Modifying  Air  Force  Aircraft 

-  Maintains  sound  airworthiness  baseline 

-  Documents  changes  to  airworthiness  criteria,  requirements  and 
methods  of  compliance 

-  Documents  seams  between  FAA  approved  areas  and  military 
approved  areas. 

-  Transient  document  -  folded  into  TACC 
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ACC  #4 


•  Airworthiness  Certification  Circular  #4  - 
Certification  Basis 

-  Issued  1 1  Mar  04 

-  Primarily  aimed  new  aircraft,  therefore  TACCs 

-  Calls  out  MIL-HDBK-514  and  MIL-HDBK-5 1 6B 
Enhanced 

-  Refers  to  FAA  Type  Certification  Basis  for 
Commercial  Derivative  Aircraft 

•  “FAA  certification  is  a  valid  certification  approach  for  criteria 
completely  satisfied  by  FAA  type  certification  on  commercial 
derivative  aircraft” 


Completion  of  TACCs  for  CDA«^f 

For  the  TACC  boiler  plate  such  as  the  cover  and  the 
airplane  description  follow  the  guidance  right  out  of  the 
HDBK  -  explain  your  system 

For  the  certification  basis  download  it  from  the  FAA 
Regulatory  and  Guidance  Library  (RGL). 

-  http://rql.faa.gov/ 

-  Select  “Type  Certification  Data  Sheets  (Make/Model)”  at  the 
bottom  of  the  lower  right  hand  column 

-  Search  for  your  aircraft’s  data  sheet. 

•  The  search  engine  works  well  for  common  civil  names:  172,  747- 
400,  G-V,  King  Air,  and  some  military  names:  KC-10A,  C-12A 

Use  the  data  sheet  to  verify  serial  numbers  and  the  civil 
certification  basis  of  the  aircraft 

-  General  Amendment  level  is  needed,  but  all  the  specific  sub 
paragraphs  don’t  add  much  value  to  the  TACC.  The  contractors 
will  work  the  details  with  the  FAA  anyway. 

•  CAR3b  through  Amnd  15,  Part  21  up  to  Amendment  21-42,  Part  23 
incldng  Amdmnt  23-56,  Part  25  through  Amendment  25-86,  etc.  12 


TACC/MACC  Section  4 


•  Section  4  includes  Tools,  Quality,  Systems 
Engineering  and  Manufacturing,  Tech  Manuals 
and  Config  -  crossing  over  all  disciplines 

•  The  basic  rule  applies,  but  in  the  manufacturing 
section,  care  must  be  taken 

-  When  parts  are  installed  on  a  CDA,  they  need  to  be 
produced  under  an  approved  production  system. 

-  All  parts  covered  by  FAA  certification,  need  to  be 
produced  under  an  FAA  approved  production  system. 

-  If  any  non  FAA  approved  parts  are  installed,  don’t  let 
them  fall  through  the  crack.  They  need  to  be 
produced  under  a  military  approved  production 
system. 
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Picking  the  Standards 


•  Once  the  applicable  criteria  have  been  identified  the 
“tabs”  corresponding  to  chapters  in  MIL-HDBK-516  need 
to  be  addressed 

•  The  basic  rule  for  cases  where  the  criteria  is  met  by  FAA 
certification  or  an  approved  FAA  process  is: 

-  For  Standards  where  FAA  Cert  covers  the  criteria,  an  acceptable 
entry  for  the  Standard  is  “FAA  Certification  (or  approved 
process)” 

-  An  acceptable  entry  for  methods  of  compliance  is  “Inspection  of 
the  FAA  approving  document” 

•  This  is  the  top  sheet  of  the  STC,  TC,  TSO,  or  what  ever  the  FAA 
process  relies  on  to  document  the  approval 

•  In  some  cases  it  may  be  a  FAA  Form  337,  or  even  a  Logbook  entry, 
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i§  r 

If  you  have  non  compliant  items 


•  Policy  relies  on  limiting  the  non-FAA 
approved  items  (Para.  2  in  both  62-4  and 
62-5) 

•  Non  conforming  items  will  be  listed  on 
FAA  Form  8130-2  “Conformity  Certificate  - 
Military  Aircraft” 

-  Standards  for  Non  Conforming  items  must  be 
“measurable  parameters”  (ACC  #4) 
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If  you  don’t  get  FAA  Certification ...  w 

•  The  Standard  and  Method  of  Compliance  for  the  FAA 
compliant  areas  of  the  aircraft  are  still  the  same  i.e, 
proven  by  the  FAA  process 

•  For  62-4  Aircraft:  non-FAA  compliant  areas  must  “meet 
or  exceed  civil  airworthiness  standards”  (AFPD  62-4, 
Paragraph  2) 

•  For  62-5  Aircraft:  “use  the  FARs  to  baseline 
requirements  where  ever  practicable”,  and  “the  SM  may 
use  Air  Force  modification  procedures  when  the  FARs 
are  inappropriate”  (AFPD  62-5,  Paragraph  2) 

IT  IS  EXPECTED  THE  STANDARDS  FOR  THE  8130-2  ITEMS  WILL  MEET  OR 
EXCEED  FAA  REQUIREMENTS  (62-4)  OR  MEET  MILITARY  REQUIREMENTS  (62- 
5)  -  IN  EITHER  CASE  YOU  NEED  TO  IDENTIFY  WHAT  THE  AIRWORTHINESS 
REQUIREMENTS  FOR  THE  8130-2  ITEMS  ARE,  AND  HOW  THEY  WILL  BE  MET 
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New  FAA  Order  81 10.101 

Type  Certification  Procedures  for  Military  Commercial 

Derivative  Aircraft 


•  New  FAA  Order:  Effective  September  7,  2007 

•  Published  and  Available  on  FAA  Regulatory  and  Guidance  Library 
Under  “Orders  and  Notices” 

•  Defines  Role  of  the  FAA  Military  Certification  and  Procedures  for  All 
Type  Certification  Approval  for  Military  Conversion/Modification  of 
Commercial  Aircraft 

•  Contains  FAA  Procedures,  Guidance,  and  Policy-  Essential  for 
Military  Program  Offices  and  Contractors  Pursuing  Commercial 
Derivative  Programs 

•  Contact  the  FAA  MCO  for  More  Information 


Managing  the  Seam  between  Civil  and  Military 
Airworthiness  Approvals  on  Commercial  Derivative 

Aircraft 

•  Issue:  Military  CDA  are  being  modified  with  FAA  issued  design 
approvals,  but  items  waived  from  FAA  certification  are  often 
incorporated  without  military  technical  oversight 

•  Issue:  Military  CDA  are  being  modified  using  “psuedo”  FAA  Field 
Approval  Process 

•  Issue:  Military  Operators  and  Contractors  are  operating  numerous 
commercial,  military  organic,  and  foreign  aircraft  with  FAA 
experimental  airworthiness  certificates 

•  Issue:  Contractors/Logistics  Suppliers  may  be  providing  parts  to 
military  CDA  which  are  not  FAA  approved  parts 


Recognize  Issues  and  Manage  Airworthiness  Seams 
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Issue:  What  process  are  you  using  to  ensure  items 
waived  from  civil  type  certification  (FAA  Form  8130-2) 
receive  military  airworthiness  approval? 


•  FAA  type  certification  is  a  closed  loop  airworthiness  process  for  civil  aircraft- 
it  requires  all  modifications  and  installed  equipment  meet  airworthiness 
standards  defined  by  civil  regulations 

•  Installed  equipment  cannot  be  ‘waived’  from  FAA  approval  on  civil  aircraft 
and  the  aircraft  allowed  to  operate  with  a  standard  (unrestricted) 
airworthiness  certificate 

•  The  two  methods  for  obtaining  FAA  approval  are 

-  the  structured  type  certification  process  with  technical  oversight  (usually  multiple 
aircraft  applications),  and 

-  the  FAA  field  approval  process  using  qualified  field  personnel  and  airworthiness 
inspectors  (for  specific  single  aircraft  only) 

•  NOTE:  No  FAA  personnel  or  FAA  designees  can  sign  a  field  approval  on  an 
aircraft  under  foreign  or  military  registration-  will  discuss  later 


Issue:  What  process  are  you  using  to  ensure  items 
waived  from  civil  type  certification  (FAA  Form  8130-31 
formerly  8130-2)  receive  military  airworthiness  approval? 


•  Items  which  have  been  identified  on  the  existing  FAA  Form  8130-31  “Military 
Statement  of  Conformity”  are  not  certified  and  have  not  been  examined  by 
the  FAA  for  airworthiness  (recognizing  that  in  some  cases  civil  certification 
is  not  possible) 

•  As  military  CDA  are  often  procured  with  intent  that  FAA  certification  will 
satisfy  airworthiness  approval  criteria,  there  is  often  no  plan  presented  for 
military  qualification/approval  of  waived  items 

•  Your  military  Program  Office  may  or  may  not  have  the  cognizance  or 
technical  resources  to  review  or  conduct  the  airworthiness  approval 

•  Late  FAA  certification  issues,  poor  program  planning,  or  intentional 
contractor  strategy  may  mean  delays  and  $$$-  your  Program  Office’s 
nightmare.  It  may  SEEM  appropriate  to  accept  system  or  component  as  a 
waived  item  and  accept  contractor  oversight  approval. 


Issue:  What  process  are  you  using  to  ensure  items 
waived  from  civil  type  certification  (FAA  Form  8130-31) 
receive  military  airworthiness  approval? 

•  First,  consider  certifying  the  modifications  to  the  aircraft  using  civil 
airworthiness  standards  to  the  maximum  extent  practical 


•  FAA  MCO  has  incorporated  guidance  into  New  FAA  Order  8110-101 
for  Type  Certification  of  Military  Commercial  Derivative  Aircraft 
which  allow  various  levels  of  certitude  to  maximize  use  of  civil 
standards 

•  These  levels  of  certitude  are  designed  to  ensure  that  modifications 
follow  a  closed  loop  process,  type  design  presented  for  certification 
comply  with  all  applicable  FAA  airworthiness  standards,  and  provide 
clear  definition  of  the  civil  and  military  airworthiness  seam 
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Levels  of  FAA  Approval  for  Military  CDA 


*  Full  Approval.  The  requirements  for  military  modifications  and  associated 
systems  and  equipment  that  meet  full  FAA  approval  are: 

-  Must  meet  the  same  requirements  for  a  commercial  modification  to  a 
civil  aircraft.  Include  type  design  data,  compliance  substantiation, 
airplane  flight  manual  supplements,  maintenance  and  continued 
airworthiness  documentation. 

-  Meet  all  applicable  airworthiness  regulations. 

-  The  installation  is  compatible  and  eligible  for  use  on  a  civil  aircraft  of 
same  type  without  special  restrictions  or  limitations. 
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Levels  of  FAA  Approval  for  Military  CDA 


•  Installation  Approval:  Military  Use  Only 

The  requirements  for  military  modifications  and  associated  systems  and  equipment 
that  meet  Military  Use  Only  are: 

-  Must  meet  the  same  requirements  as  for  a  commercial  modification  to  a  civil 

aircraft.  Include  type  design  data,  system  and  safety  analysis,  software  validation, 
compliance  substantiation,  airplane  flight  manual  supplements,  maintenance  and 
continued  airworthiness  documentation. 


-  Meet  all  applicable  airworthiness  regulations. 

-  Installation  is  NOT  compatible  or  eligible  for  use  on  a  civil  aircraft  of  same  type 
without  special  restrictions  or  limitations  (authorization  for  use,  operational 
limitations,  maintenance  requirements). 

-  FAA  may  need  help  from  the  military  to  evaluate  and  determine  compliance  with 
this  type  of  equipment  because  of  the  restriction  on  civil  operation. 

-  Installation  approvals  must  have  limitations  and  restrictions  defined  on  the  type 
design  change,  such  as  the  supplemental  type  certificate  description. 

-  If  the  limitations  and  restrictions  can  be  followed,  these  installations  may  be 
legally  permissible  to  install  on  aircraft  of  civil  registry. 


Levels  of  FAA  Approval  for  Military  CDA 

Safe  Carriage  (Partial  Approval).  A  partial  approval,  signifying  that  the  military  hardware 
and  equipment  comply  with  applicable  regulations  in  a  non-functional  state.  The  requirements  are: 

-  The  FAA  examines  the  physical  aspects  of  the  installation  including  aerodynamic  effects,  structural 
provisions,  cabin  safety,  and  weight  and  balance. 

-  The  installation,  as  defined  on  the  type  design,  complies  with  regulations  and  poses  no  hazard  to  the 
aircraft. 

-  Approval  includes  any  modifications  made  to  aircraft  structure  or  systems  to  accommodate 
installation  of  the  equipment.  Approval  does  not  authorize  or  allow  the  installed  equipment  to 
operate. 

-  Equipment  must  be  disconnected  from  power  sources,  antenna  couplers,  and  other  interfaces  with 
the  aircraft  and  these  interfaces  on  aircraft  type  design  are  safely  capped  and  stowed. 

-  Cockpit  controls  that  are  not  included  as  part  of  the  type  design,  if  the  equipment  is  controlled  or  will 
interface  with  the  cockpit. 

-  The  type  design  may  incorporate  blanking  plates  or  other  means  to  show  that  the  equipment  is  not 
approved  for  function,  and  cannot  be  enabled  or  operated  from  the  cockpit. 

-  Maintenance  covers  only  that  required  for  aircraft  provisions  (structure,  mounts,  wiring,  etc.) 
removal,  and  physical  attachment  for  securing  equipment  to  the  aircraft. 

-  “Safe  Carriage”  approvals  cannot  be  extended  to  weapons,  pyrotechnics,  or  any  other  hazardous 
materials  that  would  otherwise  be  prohibited  from  carriage  on  a  commercial  aircraft. 

-  The  receiving  military  airworthiness  authority  is  responsible  for  design  approval,  equipment 
qualification,  system  integration,  compatibility,  system  architecture,  functionality,  and  interface  with 
aircraft  systems,  operation,  and  airworthiness  approval  for  the  installed  equipment. 


Levels  of  FAA  Approval 


•  Provisions  Only  (Partial  Approval). 


The  FAA  may  also  work  with  the  applicant  and  the  military  to  define  “Provisions  Only” 
approvals  to  support  modifications  to  the  MCDA  to  receive  subsequent  military 
equipment.  Provisions  Only  approvals  are  not  on-board  installation  approvals  for  the 
military  equipment.  Provisions  Only  approvals  assess  and  approve  aircraft  structure, 
design  characteristics,  or  system  capabilities  to  handle  defined  and  predetermined 
structural  loads,  interface  or  attachment  provisions,  and  electrical  power  requirements. 
The  requirements  for  Provision  Only  approvals  are: 


-  Provide  evidence  to  support  relevant  compliance  findings  with  applicable  civil 
regulations. 

-  Address  approvals  in  the  airplane  flight  manual  and  instructions  for  continued 
airworthiness  well  enough  to  operate  and  maintain  the  FAA  approved  type  design 
configuration. 

-  Include  the  specific  criteria  for  which  the  provisions  are  approved  on  the 
description  of  the  type  design  change,  or  reference  a  document  that  establishes  all 
interface  points  and  design  limits. 

-  Ensure  that  the  receiving  military  airworthiness  authority  is  responsible  for  further 
modification  to  the  aircraft  using  the  approved  provisions  criteria. 
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Issue:  Is  your  military  operator  accepting  a  ‘pseudo’  FAA 
Field  Approval  process  for  CDA  modifications? 

What  Is  a  Field  Approval? 

A  field  approval  is  FAA  method  for  approval  of  a  major  repair  or  alteration  made  to  a 
single  civil  aircraft,  documented  as  part  of  the  FAA  Form  337  Return  to  Service 

FAA  field  approvals  on  U.S.  registered  civil  aircraft  are  conducted  under  oversight  and 
signoff  of  an  FAA  Flight  Standards  Inspector;  field  approvals  are  accomplished  for 
approval  of  major  repairs  and  alterations  on  an  individual  aircraft 

Technical  data  for  substantiation  of  major  repairs  or  alterations  may  be  approved  by 
authorized  Designated  Engineering  Representatives  for  a  aircraft  of  particular  type 
design,  but  still  require  field  approval  for  installation 

The  FAA,  FAA  employees,  or  FAA  designees  are  not  authorized  to  signoff  field  approvals 
on  foreign  registered  or  military  aircraft 

-  Exceptions-  foreign  civil  authorities  with  provisions  in  bilateral  (e.g.  Canada),  and  by 
request  of  foreign  civil  authority  for  their  aircraft  in  the  U.S. 

FAA  Order  8300.10  provides  mandatory  instructions  and  guidance  for  FAA  personnel 
regarding  execution  of  field  approvals-  including  requirements  for  technical  involvement 
by  FAA  ACO  engineering  as  complexity  increases 

Modifications  which  go  beyond  the  criteria  for  field  approvals  on  civil  aircraft  must  obtain 
type  certification  approval 


The  FAA,  FAA  personnel,  or  FAA  designees  have  no  authority  to  approve  majj>r  repairs  or 
alterations  to  an  aircraft  under  military  registration 


Issue:  Is  your  military  operator  accepting  a  ‘pseudo’  FAA 
Field  Approval  process  for  CDA  modifications? 


•  Some  contractors  have  proposed  following  the  “FAA  Field  Approval 
Process”  for  modifications  to  military  commercial  derivative  aircraft  as  a 
method  for  airworthiness  approval 

•  Some  FAA  Inspectors  and  FAA  designees,  with  good  intention,  have 
attempted  to  support  this  in  an  effort  to  assist  the  operator 

•  WARNING:  Proceed  with  risk  and  extreme  caution.  This  may  be  a  viable 
solution  in  some  circumstances,  but  has  a  history  of  problems.  Typically: 

-  Multiple  aircraft  modified  with  inadequate  documentation  of  type 
design/compliance,  inadequate  testing  and/or  substantiation 

-  Complex  modifications  and  system  integrations  which  go  far  beyond  the  scope  of 
what  would  be  allowed  by  FAA  field  approval 

-  Often  proposed  or  implemented  when  schedule/mission  need  prevents  appropriate 
procedures  from  being  followed 

-  No  oversight  or  signoff  by  FAA;  has  led  to  abuse 


Issue:  Military  Users  and  Contractors  are  operating 
numerous  commercial  derivative,  military  organic,  and 
foreign  commercial/military  aircraft  with  N  number  and 
experimental  airworthiness  certificates 

•  Anyone  with  proof  of  ownership  can  apply  and  obtain  FAA  registration  for  any  type  of 
aircraft-  registration  is  not  authorization  to  fly 

•  Civil  registered  aircraft  require  a  standard  FAA  airworthiness  certificate  to  operate  in 
national  airspace-  civil  aircraft  which  may  not,  or  cannot,  comply  with  civil 
airworthiness  standards  may  be  issued  experimental  airworthiness  certificate  for 
specific  purposes 

•  Experimental  airworthiness  certificates  require  operator  to  attest  to  airworthiness  of 
aircraft,  aircraft  may  be  inspected,  but  type  design  is  not  reviewed,  approved,  or 
certified  for  airworthiness  by  the  FAA 

•  Experimental  airworthiness  are  issued  for  specific  purposes  defined  by  CFR  Title  14 
Part  21.191  with  appropriate  restrictions  and  operating  limitations  and  must  be 
renewed  every  year  (R&D,  Show  Compliance,  Crew  Training,  Exhibition,  Air  Racing, 
Market  Survey,  Amateur  Built,  Primary  Kit  Built,  and  Light  Sport) 


Aircraft  operated  for  Public  Use  operations  do  not  require  a  civil  airworthiness 
certificate  and  are  the  responsibility  of  the  agency  directing  their  operations. 


Issue:  Military  Users  and  Contractors  are  operating 
numerous  commercial  derivative,  military  organic,  and 
foreign  commercial/military  aircraft  with  N  number  and 
experimental  airworthiness  certificates 

•  What  happens  when  operator  of  the  civil  registered  experimental  aircraft 
declares  the  aircraft  “public  use”? 

•  The  aircraft  becomes  exempt  from  compliance  with  FAA  civil  airworthiness 
regulations  and  limitations  of  the  experimental  certificate 

•  The  responsibility  for  configuration  oversight,  safety,  maintenance,  and  risk 
management  are  assumed  by  the  government  agency  operating,  or 
contracting  for  use  of,  the  aircraft 

•  If  a  military  operator,  or  contractor,  is  operating  the  aircraft  on  behalf  of  an 
armed  service,  the  Armed  Service  is  the  responsible  airworthiness  authority 

•  If  your  military  operator  or  contractor  says  their  “public  use”  aircraft 
operations  are  oversighted  by  the  FAA,  they  are  WRONG 
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Issue:  Military  Users  and  Contractors  are  operating 
numerous  commercial  derivative,  military  organic,  and 
foreign  commercial/military  aircraft  with  N  number  and 
experimental  airworthiness  certificates 

•  Does  your  armed  service  have  a  system  in  place  which  monitors  these 
experimental  public  use  aircraft?  Examples: 

-  Special  Operations  Groups,  Security,  Intelligence,  Covert  or  Clandestine  Operations 

-  Target  Towing,  Adversary  Aircraft,  Foreign  Training  Assistance 

-  Research  and  Development,  Special  Temporary  Need  Leases,  Etc. 

•  Do  you  know  how  many  and  which  aircraft  are  operating  in  this  capacity  for 
your  service? 

•  Does  your  airworthiness  authority  monitor  configuration,  operational  safety, 
maintenance,  or  risk  management  for  these  aircraft? 

•  Do  you  have  a  system  which  requires  flight  releases  or  airworthiness 
oversight  for  these  aircraft-  owned,  leased,  or  contracted  service? 
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Issue:  Contractors/Logistics  Suppliers  may  be  providing 
parts  to  military  CDA  which  are  not  FAA  approved  parts 


•  Once  FAA  certifies  type  design,  replacement  parts  sold  for  a  type  certificated 
product  require  manufacture  under  some  kind  of  FAA  Production  Approval. 
Examples: 

-  Production  Certificate 

-  Parts  Manufacturer  Approval  (PMA) 

-  Technical  Standard  Order  Authorization  (TSOA)  [design  and  production  approval] 

-  Exception:  Owner/Operator  Produced  Parts  (inspected/approved  at  installation,  not 
for  sale) 

-  Other  Exceptions:  parts  manufactured  under  type  certificate  only  (require  direct 
inspection  by  FAA),  Repair  Station  fabricated  and  inspected/approved  on-site  (not 
for  sale) 

•  But,  all  FAA  approved  parts  are  manufactured  under  some  kind  of  FAA 
approved  quality  system  or  direct  inspection 

•  Processes,  identification,  and  marking  of  approved  parts  are  controlled  in  the 
civil  airworthiness  system,  and  for  international  export 


Issue:  Contractors/Logistics  Suppliers  may  be 
providing  parts  to  military  CDA  which  are  not  FAA 

approved  parts 

•  Number  of  cases  identified  where  military  contractor/supplier  has  not 
provided  approved  replacement  parts  for  military  CDA  Examples: 

-  Detail  parts  direct  shipped  from  OEM  subcontractor,  bypassing  FAA  approved 
Production  Inspection  System 

-  STC  holder  or  supplier  fails  to  obtain  Parts  Manufacturer  Approval  (PMA) 

-  TSOA  components  installed  without  installation  approval 

-  Contractor  claimed  produced  as  Owner/Operator  Produced  Parts  for  military,  but 
cannot  be  approved/inspected  in  compliance  with  Part  43  requirements 

•  FAA  vigorously  enforces  and  prosecutes  violators  on  civil  aircraft  under  FAA 
Suspected  Unapproved  Parts  (SUPS)  program 

•  FAA  enforcement  difficult  or  impossible  on  military  CDA  when  parts  leave 
civil  system,  maintenance  occurs  outside  civil  repair  stations,  no  review  of 
aircraft  logbooks,  no  access  to  billing/shipping  records 
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Issue:  Contractors/Logistics  Suppliers  may  be 
providing  parts  to  military  CDA  which  are  not  FAA 

approved  parts 

•  Military  Airworthiness  Concerns  for  Unapproved  Parts  on  CDA 

-  Parts  may  or  may  not  conform  to  type  design,  no  FAA  inspection 

-  Military  relying  totally  on  contractor  system  (?),  with  no  government  oversight  of 
manufacturing  or  inspection  procedures 

-  Discovery  of  Unapproved  parts  on  military  CDA  could  exclude  eligibility  for  parts 
pooling  with  civil  fleet 

-  If  discovered,  flight  critical  or  safety  related  impacts  often  difficult  to  determine 

•  FAA  has  taken  enforcement  action  on  unapproved  parts  supplied  to  the 
military-  when  civil  investigation  yielded  enough  evidence  to  prosecute 

•  FAA  has  also  assisted  military  in  pursuing  contract  fraud  investigations 
when  contract  specified  supply  of  FAA  approved  parts  for  military  CDA 

•  Although  civil  regulations  require  compliance  for  parts  on  type  certificated 
products  under  civil,  foreign,  or  military  registration,  MCO  recommends 
requirement  be  clearly  incorporated  in  military  procurement/logistics  support 
contracts 
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CONCLUSIONS 


•  By  the  USAF  and  the  FAA  working  in 
partnership  the  integration  of  military  and 
civil  equipment  can  be  safely 
accomplished  on  Commercial  Derivative 
Aircraft 

•  Care  is  required  to  avoid  the  seams 
between  the  two  safety  systems 
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QUESTIONS 
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The  greatest  benefits  of  globalization  will  accrue  to  countries  and  groups  that 
can  access  and  adopt  new  technologies. 

CIA’s  National  Intelligence  Council 
Mapping  the  Global  Future ,  2004 
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References: 

Tellis,  Ashley  J,  Janice  Bially,  Christopher  Layne  and  Melissa  McPherson,  Measuring  National  Power  in 
the  Postindustrial  Age,  RAND  Report  MR  1110,  2000,  ISBN  0-8330-2792-1 

Lehman,  Ronald  F.,  The  Center  for  Global  Security  Research, 
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Innovation  Equation 
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Innovation  =  f(  Invention ,  Militarization/Commercialization,  Diffusion  ) 


Definitions 
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Invention  covers  all  efforts  aimed  at  creating  new  ideas  and 
getting  them  to  work. 

Militarization/Commercialization  (exploitation)  includes  all  stages 
of  commercial  development,  application,  and  transfer  [transition], 
including  the  focusing  of  ideas  or  inventions  towards  specific 
objectives.  Commercialization  also  includes  evaluating  those 
objectives  and  transferring  research  and/or  development  results 
for  eventual  broad-based  utilization. 

Diffusion  is  the  process  in  which  an  innovation  is  communicated 
through  certain  channels  over  time  (and  adopted)  among  the 
members  of  a  social  system. 


Reference: 

Michael  E.  McGrath,  Product  Strategy  for  High  Tech-Technology  Companies:  How  to  achieve  growth,  competitive 
advantage,  and  increased  profits,  Richard  Irwin,  Inc.,  1995,  p.  217. 

Roberts,  Edward  B.,  Generating  Technological  Innovation,  New  York,  Oxford  University  Press,  1987,  p.  3. 
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Innovation  Strategies 

(Invention  and  Commercialization) 
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>  OPPORTUNITY  DRIVEN  (Requirements  Pull) 

1.  Identify  general  opportunities  by  listening  to  specific  customer  needs. 

2.  Identify  opportunity  by  generalizing  a  solution  to  a  specific  problem. 

>  PREDICTION  DRIVEN  (Technology  Push) 

3.  Identify  opportunities  based  on  the  declining  cost  of  technology. 

4.  Identify  new  applications  for  emerging  technology. 

5.  Look  for  opportunities  at  the  intersection  multiple  emerging 
technologies. 

>  TECHNOLOGY  DRIVEN  (Science  and  Technology,  S&T) 

6.  Efficiently  search  for  solutions  to  perceived  problems/opportunities. 

7.  Stumble  over  a  technology  breakthrough. 

>  SYSTEM  DRIVEN  (Science  and  Technology,  S&T) 

8.  Organize  design  components  together  in  a  new  way,  using  many  core 
design  concepts  in  a  new  architecture. 

9.  Ljnk  significant  resources  from  different  disciplines,  businesses  and 
government  agencjes. 

Reference:  Adapted  from  Michael  E.  McGrath,  Product  Strategy  for  High  Tech-Technology  Companies:  How  to 
achieve  growth,  competitive  advantage,  and  increased  profits,  Richard  Irwin,  Inc.,  1995,  p.  217. 
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Rapid  User  Focused 
Invention  and  Militarization 
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CONOPS 

Affordability  from  the  intersection  of  existing  military  technologies: 

•  Reliability 

•  Maintainability 

•  Training 

•  Support  Infrastructure 

•  Compatibility 

•  Familiarity 

•  Reduced  Technology ,  Cost  and  Schedule  Risk 

•  Economies  of  Scale  from  Multiple  Applications 
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User  Focused  Example:  Spartan 
Scout  Unmanned  Surface  Vessel 
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Diffusion  of  Technology 
Has  Accelerated 
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Telephone 


The  Spread  of  Products  into  American  Households 
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Years  Since  Product  Invented 


Sources:  Bureau  of  the  Census;  The  World  Almanac;  Cellular  Telecommunications  Industry  Association 


Rapid  diffusion  of  innovation  is  the  challenge . 
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Technology  Readiness  Levels  vs. 
Attributes  for  the  Diffusion  of  Innovation 
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The  TRL  system  was  adopted  by  the  NASA  space  program  for 
project  tracking  and  management  and  was  incorporated  in  NASA 's 
1991  Integrated  Technology  Plan. 

There  are  nine  different  TRLs: 

1.  Basic  principles  observed  and  reported 

2.  Technology  concept/application  formulated 

3.  Critical  function  proof  of  concept 

4.  Component  validation  in  laboratory 

5.  Component  validation  in  relevant  environment 

6.  System  prototype  in  relevant  environment 

7.  System  prototype  in  operational  environment 

8.  Actual  system  completed  and  qualified  via  T&E 

9.  System  proven  through  mission  operation 


System  Test,  Launch 

&  Operations 

System/Sirbsyslenn 

Development 

Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


TRL  9 
TRL  8 

TRL  7 


TRL  4 

TRL  3 


Reference: 


Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
(MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition 
Programs,  DoD  5000.2-R,  5  April  2002 


Basic  Technology 
Research 
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Perceived  Attributes  for  ib 
the  Diffusion  of  Innovation 

1.  Relative  advantage,  the  degree  to  which  an  innovation  is 
communicated  as  being  more  cost  effective  than  the 
innovation  it  supersedes 

2.  Effectiveness ,  the  degree  to  which  an  innovation  is 
communicated  as  being  relatively  more  capable  in 
achieving  an  ideal  end  state 

3.  Observability,  the  degree  to  which  the  results  of  an 
innovation  are  communicated  as  being  visible  to  others 

4.  Tnalability,  the  degree  to  which  an  innovation  is 
communicated  to  allow  experimentation  on  a  limited  basis 

5.  Complexity,  the  degree  to  which  an  innovation  is 
communicated  as  being  relatively  difficult  to  use 

6.  Compatibility,  the  degree  to  which  an  innovation  is 
communicated  as  being  consistent  with  past  experiences, 
existing  practices,  and  needs  of  potential  adopters 


Reference: 

Rogers,  Everett  M.,  Diffusion  of  Innovation,  Free  Press,  2003. 

Dearing,  James,  “An  Exploratory  Tool  for  Predicting  Adoption  Decisions,”  Science 
Communications,  Vol.  16,  No.  1,  September  1994,  43-57. 
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Perceived  Attributes  for  ifii 
the  Diffusion  of  Innovation 

7.  Reliability,  the  degree  to  which  an  innovation  is 
communicated  as  being  consistent  in  its  results 

8.  Divisibility,  the  degree  to  which  an  innovation  is 
communicated  as  allowing  incremental  implementation  of 
its  components 

9.  Applicability,  the  degree  to  which  an  innovation  is 
communicated  as  having  more  than  one  use,  or  having  use 
in  more  than  one  context 

10.  Commutability,  the  degree  to  which  an  innovation  is 
communicated  as  exhibiting  a  complementary  relationship 
with  other  innovations 

11.  Radicalness,  the  degree  to  which  an  innovation  is 
communicated  as  being  different  from  existing  innovations 

The  innovation-decision  process  is  essentially  an  information  seeking  and 
processing  activity  in  which  an  individual  is  motivated  to  reduce  uncertainty 
about  the  advantages  and  disadvantages  of  the  innovation. 

Reference: 

Rogers,  Everett  M.,  Diffusion  of  Innovation,  Free  Press,  2003. 

Dearing,  James,  “An  Exploratory  Tool  for  Predicting  Adoption  Decisions,”  Science 

Communications,  Vol.  16,  No.  1,  September  1994,  43-57. 
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Diffusion  Assessment  Example  ■ 


Attribute 

Intent 

or 

Perce 

ption 

(1-7) 

Comment 

Applicability 

1 

Highly  Applicable,  high  utility 

Reliability 

6 

Not  strongly  communicated 

Compatibility 

1 

Compatible  with  existing  concepts  of  operation 

Divisibility 

2 

Alternate  options  communicated 

Radicalness  (reverse  coded) 

5 

Communicated  effort  as  being  different 

Complexity  (reverse  coded) 

2 

Not  highly  complex  (context  dependent) 

Trialability 

1 

Use  of  systems  in  experimentation  strongly  emphasized 

Observability 

4 

Not  strongly  communicated 

Effectiveness 

2 

Analysis  used  to  determine  performance 

Commutuality 

1 

Complementary  nature  strongly  articulated 

Economic  Advantage 

5 

Not  strongly  communicated 

Total 

30 

Avg 

2.73 

Reference: 


Dearing,  James,  “An  Exploratory  Tool  for  Predicting  Adoption  Decisions,”  Science 
Communications,  Vol.  16,  No.  1,  September  1994,  43-57. 
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Innovation  Strategies  for  'rtn 
Affordable  Readiness  Summary 

>  Respond  to  emerging  threats  with  innovation: 

-  Invention 

-  Militarization/Commercialization 

-  Diffusion 

>  Develop  affordable  products  rapidly  by: 

-  Identifying  opportunity  driven  concepts  via  specific  customer  needs 

-  Developing  prediction  driven  products  based  on  the  intersection  of 
technologies 

>  Focus  on  diffusion  attributes: 

-  Incorporate  readiness  levels  as  part  of  diffusion  attributes 

-  Reduce  uncertainty  for  end  users  through  the  11  attributes 

-  Assess  innovative  concepts  and  products  to  determine  the  potential 
for  diffusion 
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NEWPORT 


Our  Main  Thing  is  working  with  Industry,  Academia, 
and  Navy  Labs  to  deliver  solutions  to  the  Warfighter 
where  and  when  they  are  needed. 


MISSION 


Conducts  a  full  spectrum  program  of  research,  development,  engineering,  and  test  &  evaluation  directed 
toward  underwater  sensors  and  sonar  systems  applicable  to  all  platforms  as  well  as  off-board  distributed 
and  unmanned  systems,  with  equal  emphasis  on  technology  base,  advanced  development,  full-scale 
development,  and  in-service  engineering,  supportability  and  life-cycle  hardware  and  software  support. 
Focuses  on  analysis,  definition,  development,  system  engineering,  integration  and  testing.  The  mission 
focus  is  on  all  aspects  of  Undersea  Warfare  (USW)  and  associated  areas  of  the  Global  War  on  Terrorism 
(GWOT). 


Products  and  Capabilities 


V  Active  and  Passive  Acoustic  Systems 

V  Environmental  Acoustic  Technology  and 
Systems 

S  Hull-mounted,  Fixed,  and  Towed  Sonar 
Systems 

■S  Offboard  Sensors  and  SONAR  Systems, 
Including  Distributed  Systems 

V  Human  Systems  Integration  for  Manned  & 
Unmanned  Systems 

V  Sonar  Trainer  Systems 

V  Transducers,  Materials,  Measurements,  and 
Standards 


■S  Underwater  Acoustic  Communications 
Systems 

■S  Underwater  Off-Board  Sensors  and  SONAR 
Systems 

V  Underwater  Non- Acoustic  and  Environmental 
Sensors 

V  Unmanned  Vehicle  Sensors  and  SONAR 
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Systems  Engineers  Need  A  Depth  of 

Knowledge 


TPM  PROCESS  &  KNOWLEDGE 


SYSTEM  ENGINEERING  KNOWLEDGE 
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Good  Systems  Engineers 
Demonstrate 


■mi 


□  Technical  Expertise  and  Competence 

□  Solid  Engineering  Design  Skills 

□  Willingness  to  Take  on  Leadership  and 
Responsibility  Roles 

□  Effectiveness  Working  in  Dynamic  Team  and 
Interdisciplinary  Environments 

□  Excellent  Communications  Skills 


Not  all  engineers  display  the  desire  and 

aptitude 

to  be  a  Systems  Engineer. 

They  are  not  merely  “made’...  innate 

qualities  and 
interest  are  necessary. 
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Typical  System  Engineering  Career 
Development  Timeline 


Career  Stage 

Elapsed  Time 

Concurrent 

Age 

New  Hire 

23 

Develop  Technical 

Experience  in  Engineering 
Discipline 

5-8  years 

28 

T&E  At-Sea 
Experience/System 
Installation/System  Delivery 

1  -  2  years 

29 

Rotational  Assignments: 

•  Other  Engineering  Specialties 

•  Operational  Experience 

•  On-Site  Assignments  (l.e. 

Field  Team) 

•  International  Experience 

1  -  2  years 

30 

Technical  Program 
Manager/Task  Leader 

2  years 

32 

System  Engineer 

2  years 

34 

TOTAL 

11-16  years 

34  -39 

Development  of  Systems  Engineers  NDIA  OCT07 
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Combined  Surface  Ship  USW,  Submarine  Sonar  and 
Distributed  USW  Sensor  Systems  Tech  Warrant  Pyramid 


NEWPORT 


With  At  Risk  Positions 


NUWC/N 


SSC/NF 


Analysis 

□  Acoustic  Analysis 

□  Sonar  Performance 
Analysis 

□  Environmental  Modeling 
&  Assessment 

□  Hydrodynamic  Data 
Analysis 

□  Mission  Effectiveness 
Analysis 

□  Signature  Analysis 

□  Sonar/Platform/Battle- 
group  Engineering 
Assessments 


Installation 

□  OPALTS 

□  SHIPALTS 

□  TEMPALTS 


Sonar  Array 
Technology 

□  Acoustic  Measurement 

□  Acoustic  Modeling  & 


Calibration  & 
Measurement  Facilities 

□  Array  &  Sensor 
Calibration 


Modeling  and 
Simulation 

□  Finite  Element  Modeling 
&  Assessment 

□  Model  Development  & 
Verification 

□  Modeling,  Testing  & 
Qualification 

□  Sonar  SIM/STIM 

□  Structural  Acoustics 

□  Target  Physics  Modeling 
&  Research 

□  Towed  Body/HD  Models 


ILS 

□  CM 

□  COTS  Supportability 
Analysis 

□  Fleet  Technical 
Publications  Support 

□  Logistics  System  Support 

□  Operational  Guidelines  & 
Employment 

□  Training 

□  Trainers 


Signal  Processing 

□  Active  Signal  Processing 

□  Passive  Signal 
Processing 


Standards 
Acoustic  Windows 
Active  Source 
Performance 
Advanced  Arrays 
Array  Structures 
Developmental  Arrays 
Hull,  Deployed,  & 
Towed  Sensor 
Engineering 
Sonar  Projectors 


Sonar  Engineering 

□  Acoustic  Communication 
Advanced  Processing 
Advanced  Processing  Builds 
Advanced  Systems  Prototyping  &  Ev; 
Algorithm  Development 
Array  Handling  Technology 
Automation 

Data  Fusion/Contact  Management 
Detection,  Classification 
Display  Engineering 
HF  Imaging  &  ASW 
Mine  Detection  &  Avoidance  Sonar 
Multistatics 

Oceanographic  Engineering 
Optical  Application  to  Sensors 
Prototyping 
Rapid  COTS  Insertion 
Real  Time  COTS  Processing 
Recon  Docking  &  Side  Scan  Sonars 


i) 


Transducer  &  Array 
Materials 


In  Service  Engineering 

□  Array  Maintenance  & 
Repair 

□  Fleet  Introduction 

□  Fleet  Liaison 

□  Fleet  Support 

□  Production  Engineering 

□  Restoration  Engineering 

□  Systems  Supportability 


Software  Engineering 

□  Database  Development 

□  Information  Assurance 

□  Software  CM 

□  Software  Development 

□  Software  Integration  and 
Unit  test 

□  Software  Process 

□  Software  Requirements 
(development,  allocation 
&  management) 

□  System  Software 
Architecture 


Transducer  Design 
Telemetry  Technology 
Towed  Sensors 
Sphere,  Large 
Aperture/Conformal  & 
Towed  Arrays 
Fiber  Optic  Sensor 


System  Engineering  System  Engineering  (Cont’ 

□  Anti-Tamper  Engineering  □  Augmentation 

□  COTS  Engineering  □  Data  Acquisition  &  Analy|si 

□^onarPowerAm^lifier  EngineSir^nvironmental  Sensors 

^pcess  □  Test  Planning 

□  Shock  Qualification 

□  HSI  Engineering  □  System  Assessment  Tes|ti 

D  Interface  Engineering  Energy  Generation 

□  Legacy  System  Engineering 

□  Open  System  Architecture  Er!^n£$fflj^unicatlon 

□  Reliability  Engineering 

□  Requirements  Allocation 

□  Risk  Management 


ing  (in 


□  Sensor  Requirements 

□  Specifications 

□  SUBSAFE 

□  System  Requirements 

□  System  Safety 

□  TDA  Support  Technical  Failure  Analysis 


Floating  Sensors 
Bottom  Sensors 
Self  Propelled  Sensors 
Airborne  Deployment 
Shipboard  Deployment 
Launchers 


Signal  Conditioning  Micro-Electronics  FQtdt^hflology  Transition 


Sonar  Systems  Engineering 
Technology  Insertion 
Tracking  &  Localization 
Platform  Engineering 


□  USW  System  Conceptual  Development 

□  USW  System  Functional  Decomposition 

□  USW  System  Integration 

□  Testing,  Evaluation  &  Certification 

□  At-Sea  T&E 


Surface 

Personne 


30 


Dist  USW  SysQ  Q  Q  Q 
Personnel 
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■llll 


WORKFORCE  SHAPING  PLAN  SUMMARY 

•  Workforce  shaping  changes  the  potentials  and  abilities  of  the  workforce  that  can  be  applied  to  current 
Navy  problems.  Tools  for  changing  the  workforce  include 

-  Hiring:  Change  the  current  mix  of  the  workforce.  Both  external  and  internal  acquisitions  are  used  at  both  new 
and  journey  level  persons. 

-  Training:  Increasing  the  education  or  skills  of  members  of  the  organization  through  college  training ,  specific  skill 
training,  or  on  the  job  experiences. 


HIRING  MODEL 


PROFICIENCY  NEEDS 


RISK  FACTOR 

Position 

C  linen 
fly  at 
Risk 

Project 
ed  at 
Risk 

Capac 
rty  at 
Risk 

ANALYSIS 

Sonar  Performance 
Analysis 

X 

SIGNAL  PROCESSING 

Active  Signal  Processing 

X 

Passive  Signal 

Processing 

X 

SONARARRAY 

TECHNOLOGY 

Acoustic  Transducer 
Metrology  and 

Standards 

X 

Fiberoptic  Sensors 

X 

SONAR  ENGINEERING 

Sonar  Systems 

Engineering 

X 

SYSTEM  ENGINEERING 

Sonar  Power  Amplifier 
Engineering 

X 

Testing,  Evaluation  & 
Certification 

X 

At-Sea  "f&E 

X 

Mechanical 

Engineering 

Platform  Installation 
Engineering 

X 

Development  of  Systems  Engineers  NDIA  OCT07 
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Formal  Training 


□  College  Courses 

•  Tuition  for  advanced  degrees 

•  Long  term  training 

□  Commercial  Training -Short  Courses 

•  On-site  courses 

□  Special  Programs 

•  SE  and  Program  Mgmt  degrees  programs 

□  DAWIA 

•  Most  of  our  people  fall  under  SPRDE 

•  Required  level  III  certification 

•  Courses  available 
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Tailored  On-Site  Training 


5  October  200  S  WORKJN  G  D  RAFT  Vffl.5 


Sensors  and  Sonar  Systems 
Department  Systems  Engineering 
Training  Plan 

Working  With  Indus  tty.  Academia,  and  Navy  Labs  to 
Deliver  Solutions  to  the  Warfighter  Where  and  When 
They  Are  Weeded 


NKr«T-’-:  jh 


Naval  Undersea  Warfare  Center  Division 
Newport,  Rhode  Island 

...Working  Together  to  Defiver  the  Best  Solutions  Quickly 

Prepared  by,  L.  Lazar  (1541)  and  G.  Maris  (1542) 
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Title 

Status 

Introduction  to  Sys  Eng 

Complet 

e 

DoD  5000  Summary 
Presentation 

Complet 

e 

Navy  Organization  & 
Funding 

Complet 

e 

T&E  Fundamentals 

Complet 

e 

Naval  Organization  & 

CWC  Concept 

Complet 

e 

Basic  ASW 

Complet 

e 

Sonar  Fundamentals 

Complet 

e 

Sonar  Design  Trade-offs 

Complet 

e 

Acquisition  Fogistic 

Support 

Outline 

Human  System  Integration 
(HSI) 

Outline 

EQT 

In 

Process 

Title 


Status 


USW  Systems 

Overview 

Not 

Started 

Requirements 

Not 

Started 

Development  Models 

Not 

Started 

Modeling  & 
Performance 

Prediction 

Not 

Started 

Sensors  to  Displays 

Not 

Started 

Program  Management 
for  SE 

Not 

Started 

Business  Office 

Not 

Started 

Personal  Development 

Not 

Started 

HSI 

Not 

Started 

Training 

Not 

Started 

SW  Engineering 

Not 

Started 

Subsafe 

Not 

Started 

Sonar  Automation 

Not 

Started 
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Results 


□  8  Sessions  Conducted  to  Date 

□  Responses  Generally  Good 

□  Delays  Due  to  Other  Demands  On 
Presenters/Organizers  Time 

□  Some  Variability  In  Quality 

□  Requests  To  Expand  Program 
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NDIA  10th  Annual  Systems 
Engineering  Conference 


System  Engineering  Effectiveness 


A  Robust  Process  for  Resolving  Interface 
Design  Issues  in  the  Complex  Concurrent 
LCS  System  Engineering  Environment 


Mr.  William  Traganza 
Assistant  Program  Manager 
Maritime  Surveillance  Systems  (PMS  485) 

CDR  Joseph  Conway  USN  (Ret.) 

Director,  Systems  Engineering 
AMRON 

Distribution  authorized  to  U.S.  Government  agencies  and  their  contractors;  Administrative  /  operational  use  (Feb  93).  Other  requests  for  this  document 
will  be  referred  to  CNO  (N774).  Exempt  from  distribution  to  Defense  Technical  Information  Center  in  accordance  with  DoD  Directive  3200.12. 
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■  Introduction 

■  Background 

■  Impacts  of  concurrency  on  ADS  Program  (ACAT-1) 

■  Challenges  to  Traditional  SE  Process  for  handling 
concurrency 

New  SE  process  developed  to  address  problem 

■  Lessons  Learned 

■  Recommendation 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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FY08  Congressional  Language 


HAC  Committee  Report  on  Congressional  Oversight 


HR  1585  SEC.  127.  LIMITATION  ON  CONCURRENT  DESIGN  AND 
CONSTRUCTION  ON  FIRST  SHIP  OF  A  SHIPBUILDING  PROGRAM. 

(a)  In  General-  For  any  shipbuilding  program  that  is  a  major  defense  acquisition 
program  under  section  2430  of  title  10,  United  States  Code,  the  start  of  construction 
of  a  first  ship  (as  defined  in  subsection  (b))  may  not  occur  until  the  Secretary  of  the 
Navy  certifies  to  the  congressional  defense  committees  that  the  detailed  design  of 
the  ship  is  completed  and  approved  by  the  relevant  design  certification  agents,  to  a 
level  determined  by  the  Secretary  to  be  acceptable  for  commencement  of 
construction,  via  a  report  described  in  subsection  (d). 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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Traditional  Concurrency  Definition 


■  Degree  of  overlap  between  processes  of  an  acquisition 
program 

•  Development 

•  Production 


■  Structuring  the  overlap  is  a  management  judgment; 
(it  is)  a  balancing  of 

•  Risk  of  proceeding  with  certain  production  and  other 
activities  weighted  against 

•  Costs  of  delays  caused  by  work  force  inefficiency 


*  Source:  DOD  Report  on  Concurrency  in  Major  Acquisition  Programs, 
April,  1990 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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New  SoS  Concurrency  Definition 

■  Degree  of  overlap  between  interface  development  of 
two  or  more  acquisition  programs 

•  Two  LCS  configurations 

•  ADS 

■  Structuring  the  overlap  is  the  objective  of  this  paper 

•  SoS  development  is  here  to  stay 

•  Processes  needed 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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Objective 


This  paper  will  describe: 

How  PMS  485  managed  the  “SoS  concurrent”  design 
environment  of  an  ACAT-1  sensor  development  for  two 
competing  LCS  designs 

■  Lessons  Learned 

■  Recommended  SoS  SE  Process 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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ADS  History  /  Timeline 


Old  SE 
Framework 


Concept 

Exploration 

Program 

Engineering  & 

Production,  Fielding  / 

Definition  &  Risk 

Manufacturing 

Deployment,  & 

Reduction 

Development 

Operational  Support 

ACAT  II 


1992 


Recommend 
To  be  ACAT  I 


Pre-MDAP 


1994 


Jan  00 


TD 

Contract 


Sep  04  Sep  05 


Jun  00:  Redirection 


May  03:  Redirection 

A 


IQtr  3Qtr  4Qtr 
08  08  09 


NewSE 

Framework 


Concept 

Refinement 


Technology 

System  Development 

LRIP 

Production  & 

Operations 

Development 

&  Demonstration 

Deployment 

&  Support 

DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
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LCS  History  /  Timeline 


■  Dec  02  New  start  authorized  by  Congress 

■  Feb  03  RFP  for  Preliminary  Design 

■  Jul  03  Award  3  Preliminary  Designs 

■  May  04  MS  A  /  Program  initiation  /  down  select  to  2 

■  Dec  04  Detail  Design  and  Construction  Option  to  LMCO 

■  Feb  05  Start  Fabrication 

■  Oct  05  Detail  Design  and  Construction  Option  to  GD 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
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Potential  Impacts  of  Concurrency 
on  ADS  Program  (ACAT-1) 


■  Cost:  Were  ADS  reserves  for  ADS  changes  for  two  lead  ships  enough?  Did  LCS 
PM  reserve  enough  to  pay  for  changes  to  MMs  driven  by  his  unilateral  changes? 


■  Performance:  How  would  LCS  designs  constrain  ADS  KPPs? 


■  Schedule:  Would  LCS  be  ready  to  support  ADS  Test  events  and  OPEVAL  dates 
IAW  ADS’  ACAT-1  Milestones? 


■  Design  Risk:  What  changes  would  ADS  be  forced  to  implement  to  match  LCS 
design  instability  and  decisions  after  ADS  CDR? 

■  Process:  What  processes  would  LCS  PM  adopt  to  implement  CM  while  trying  to 
satisfy  the  objectives  of  “transformational  ship  acquisition”? 

■  Contractual  Liabilities:  What  impact  would  the  LCS  dual-lead  ship  acquisition 
strategy  have  on  communication  among  the  parties 


An  inter-program  coordination  process  could  mitigate  these  risks 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
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Interdependencies  of 
2ACAT-1  Programs: 
Challenges  of  SoS  Concurrency 


■  Multiple  Process: 


•  Total  ship  SE  (TSSE)  processes  vs.  traditional  industry 
SE  processes  (EIA  632,  IEEE  1220,  etc.) 

■  Multiple  Definitions: 


•  e.g.,  “ICD”:  Initial  Capabilities  Document,  Interface  Control 
Document  or  Installation  Control  Drawings 

Multiple  Interface  Controls  /  Configuration  Management: 

•  4  spiraling  /  evolving  baselines 

•  3  Configuration  Control  Boards 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 
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Interdependencies  of 
2ACAT-1  Programs: 
Challenges  of  SoS  Concurrency  (cont. 


■  Multiple  Competing  Schedules: 


•  Access  to  seaframes,  integrated  testing,  PDT  &  T,  ADS 
OPEVAL 


•  Cross  sponsor  (OPNAV)  and  Fleet  coordination 

■  Multiple  Competing  OSD  Acquisition  Program  Baselines 
(APB)  /  Reports  /  Milestones 

•  Generally  do  not  recognize  interdependencies  of 
Systems  of  Systems  development  sponsored  by  different 
OPNAV  offices 


•  APBs  do  not  anticipate  changes  forced  upon  a  program 
due  to  changes  by  others 
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Concurrent  Spiral  Development 

Touch  Points 


■ 


The  most  critical  element  of  success  was  robust  CM 
of  designs,  documents  and  processes 
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8  Element  SoS  SE 
Process  Developed 


1. 

2. 

3. 


4. 


5. 

6. 

7. 

8. 


Created  Gate  Keeper  process  and  VPO  site  (vs.  serial 
information  processing) 

Established  Cross  Program  Configuration  Management 

Developed  Interface  Requirements  Document  (needed  for  ADS 
contract  in  absence  of  a  ship  ICD.  Addendum  to  the  SPS  for 
external  interfaces) 

Developed  the  Assumptions  Process  (to  support  PDR,  CDR 
and  detail  design)  and  applied  the  BCR  process  to  changes  in 
assumptions 

Developed  Adjudication  Process  and  forum  (to  allocate 
assumptions  to  formal  documents  under  CM) 

Created  managed  Engineer-Engineer  forums 

Performed  special  studies  for  critical  issues 

Integrated  IMS 
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1.0  Gate  Keeper  (IPT) 

■  Process  created  to  shorten  and  enhance  technical 
information  exchange  in  a  competitive  acquisition 
process 

Traditional  serial  process  was  shortened  by  creating  an 
IPT  structure  called  “Gate  Keeper” 

Short  circuited  time  lag  by  providing  a  web-enabled 
Integrated  Data  Environment  (IDE) 

■  Parties  signed  non-disclosure  agreements  among  each 
other  to  limit  access  to  and  unwitting  disclosure  of 
competition  sensitive  data 

Meetings  held  monthly  and  telecons  weekly 
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Traditional  Serial 
Communications  Path 
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Gate  Keeper  CPT 


LM  or  GD  LCS 
Seaframe 
Manage  /  Design 
Team 


PEO(S)  (PMS-501) 
MSSIT 


PEO  C4I 


PEO  IWS  (IWS5) 


PEO  LMW  (PMS-420)  ASW  LCS 


Mission  Package  Lead 


PEO  LMW  (PMS-485)  APMs 


LM  ADS  Manage  /  Design  Team 


Sub-System  Design 
IPT  Leads 


SE  &  I  IPT 
Lead 


ILS  CPT 
Lead 


T  &  E  CPT 
Lead 


Gate  Keeper  Information  Librarian  (GIL) 


SE  &  I 
IPT 


T  &  E 


Sub-System  Design 
IPTs 


Acts  as  bookkeeper  and  information 
conduit  between  PMS-501  /  PMS-420 
and  PMS  485  /  ADS  Contractors 


CPT 


CPT 
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2.0  Cross  Programs  CM 

■  Four  key  elements 

•  Assign  requirements  between  Seaframe  and  Mission 
Package 

-  Identified  which  documents  and  processes  needed 

•  Identification  of  Critical  Interface  drawings  and  specs 

•  Development  and  allocation  of  key  assumptions  to 
appropriate  requirement,  drawing,  process  specification, 
etc.,  and  put  under  CM 

•  Assignment  of  lead  POCs  with  SE  competency  on  partner 
CM  forums 
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2.0  Cross  Program  CM  (cont.) 


■  ADS  SI  Team  worked  cooperatively  with  the  following 
CM  programs: 

•  501  MSSIT  CM  (via  420  and  Local  485  Representative) 

•  420  ASW  CM  (Direct  Member) 

•  501  C2  IPT  CM  (Direct  Member) 

•  Sea  Frame  CM  (via  420  and  Local  485  Representative) 
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Mission  Module  Requirements  and 
Design  Documents 


Code 

Document 

Version 

Interface  Control  Document  (ICD)  For  The  Littoral  Combat  Ship  (LCS) 
Flight  Zero  Reconfigurable  Mission  Systems 

Version  1.2 

PMS-501 

LCS  Interface  Drawing 

Per  Gate  Keeper  Drawings  List 

Littoral  Combat  Ship  Mission  Package 

Interface  Design  Specification 

Version  1.0 

Requirements  for  the  Littoral  Combat  Ship  (LCS)  Flight  0  -  Spiral  Alpha 
Anti-Submarine  Warfare  (ASW)  Mission  Package 

15  Sep  05 

LCS  Flight  0 

Mission  Package  Computing  Environment 

20  Jan  06,  Version  1.3 

PMS-420 

On-Board  ASW  Mission  Component 

Inter-Mission  Systems  /  Modules 

Interface  Requirements  Specification  (IRS) 

And  Interface  Design  Document  (IDS) 
for  the  Littoral  Combat  Ship  Flight  0 

10  Jan  06 

ASW  Mission  Package  -  MPCE  General  Purpose  Set 

Flight  0  Computing  Resources  Allocation 

14  Dec  05,  Version  3.0 

MP  ISS  Programmer's  Guide 

7  Mar  05,  Version  1.1 

Mission  Package  Infrastructure  (MPI) 

Mission  Module  (MM)  to  MPI  Interface  Requirements  Specification  (IRS) 

4  May  05,  Version 

Mission  Package  Infrastructure  (MPI) 

Mission  Module  (MM)  to  MPI  Integration  Requirements  Specification 
(Int  RS) 

Undated;  last  change  Chg  11;  30  Sep  05 
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CM  Status 


■  PMS  485  /  ADS  requested  CM  on  10  Interface 
Specifications  and  15  drawings 

•  PMS  501  /  420  provided  periodic  status 


Drawing  /  Specification  CM  Summary 
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3.0  Interface  Requirements 
Document  (IRD) 

■  Document  created  as  a  holding  repository  for  interface 
needs  of  the  ADS  system  in  the  absence  of  Installation 
Control  Drawings  or  Interface  Control  Document 

■  Began  in  FY04 


No. 

Title 

Notes 

IRD22591 

3.2.1 .8.3  Storage 
Media  Classification 
Level 

IRD22592 

LCS  shall  provide 

ADS  secure  storage 
at  the  SECRET  level 
for  all  off-line 
recording  and  archive 
storage  media 

Keep  Requirement  in  ADS  IRD. 

Rewrite  as:  ADS  shall  not  require  more  than  XX  m3  of  secure  storage  at 
a  secret  level  for  ADS  recording  and  archive  storage  media. 

Link  up  to  new  requirement  in  ASW  MP  SPS. 

Submit  BCR:  ASW  MP  shall  not  require  more  than  XX  m3  of  secure 
storage. 
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ADS  Interface  Development 

PDR  was  based  on  known  requirements  and  IRD  assumptions 

■  Key  issues  requiring  mitigating  risk  assumptions  for 
SDD  /  CDR  included: 

•  Pre-Staging  of  ADS  Strings  versus  NAVSEA  901 B  Shock 
requirements  and  Mission  Bay  space 

•  Topside  and  Mission  Bay  environmental  conditions 

•  Ability  to  maneuver  /  handle  ADS  equipment  from  all  stowage 
locations  using  shipboard  transport  equipment 

•  Compatibility  of  ADS  and  LCS  computing,  network  and 
IA  architectures 

•  ADS  RF  Link  Margin  Requirements 

PDR  designs  had  gone  as  far 
as  possible  with  existing  information 
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Extended  SE  principles  to  concurrent  program  developments 

Implementation  responsibility  was  by: 

•  MSSIT  for  PMS  501 

•  ASW  MP  SE  Group  for  PMS  420 

•  ADS  SI  IPT  /  SE&I  IPT  for  PMS  485 

New  Assumptions  were  generated  as  programs  matured 

•  ADS  update  letters  and  tables  developed 

•  Updated  sets  were  sent  to  PMS  420  and  PMS  501  after  final  ADS  PMO 
review 

■  Assumptions  were  allocated  to  appropriate  owner  for  resolution,  tracking  and 
adjudication 

■  Eventually,  these  were  incorporated  into  the  ADS  contract  and  turned  over  to 
the  ADS  prime  for  CM 

Assumptions:  Criteria  effecting  a  system  design  that  the  parties 
are  forced  to  adopt  in  the  absence  of  stated  requirement, 
specification,  procedure  or  operating  parameter  (e.g.  environment) 
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Formalized  Inter-Program  Process 


4  Nov  05 


From:  Program  Executive  Office, Littoral  and  Mine  Warfare, Maritime  Surveillance  Systems  Program  Office  (PMS  485) 
To:  Program  Executive  Office,  Ships,  Littoral  Combat  Ship  Program  Office  (PMS  501) 

Program  Executive  Office,  Littoral  and  Mine  Warfare,  Littoral  Combat  Ship  Mission  Modules  (PMS  420) 

Subj:  ADS  MISSION  MODULE  ON  LCS  ASSUMPTIONS  AND  RESPONSIBILITIES 
Enel:  (1)  ADS  on  LCS  Assumptions,  Rationale  and  Responsibilities  Matrix 


Background.  ADS  is  a  Mission  Module  (MM)  of  LCS  ASW  Mission  Package... we  have  found  that  the  LCS  Interface  Control 
Document  (ICD)did  not  have  sufficient  details  for  the  purpose  of  supporting  the  preliminary  design  of  ADS.. .pending  interface 
decisions  may  drive  changes  to  both  the  ADS  and  LCS. 

ADS  is  awarding  a  contract,  assumptions  and  responsibilities  have  to  be  made  for  these  pending  interface  decisions. 
Enclosure  (1)  provides  rationale  and  CCB  responsibility  for  each  assumption... necessary  to  allow  PMS-485’s  prime  contractor 
to  propose  a  technical  and  cost  solution  for  the  ADS  SDD  phase. 

Changes.  ADS  SDD  contract  funding  is  based  on  these  assumptions.  Only  the  responsible  CCB,  as  specified  in 
Enclosure  (1),  can  change  these  assumptions.  Changes  in  these  assumptions  could  result  in  ECPs.  The  respective  CCB,  will 
include  PMS-485  to  arrive  at  acceptable  alternatives,  funding  sources,  and/or  schedule  impact  and  resolution  of  schedule 
changes  for  those  changes  which  may  effect  ADS. 

By  Direction 
P.  F.  SEIDEL 

Copy  to: 

RADM  Hamilton,  PEO  Ships 
RDML  William  Landay,  PEO  LMW 
Ken  Montgomery,  PEO  Ships,  PMS  501 M 
Ken  Michaud,  PEO  LMW,  PMS  420 
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Category 

Total  Lines 

PMS  485 

PMS  420 

PMS  501 

Total 

96* 

20 

27 

57 

Programmatic 

15* 

3 

6 

13 

MPCE  Computing 

18* 

6 

8 

5 

Comms 

22 

4 

2 

16 

Launch  &Recovery 

24 

7 

3 

14 

Qualification  4  2  1  1 
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Overview  of  ADS  Assumptions 
(Communications  Summary) 


Section 

Title 

Requirement 

3.2.2.2 

ADS  /  LCS  Data  Telemetry 

LCS  shall  provide  4  ea  chassis  locations  (3  radio  chassis  and  1  controller  chassis)  in  the 
radio  room  for  the  ADS  radio.  Each  radio  chassis  will  conform  to  ANSI  /  EIA-310  dimensions, 
19  inches  wide  by  25  inches  deep  by  9  U  high. 

3. 2. 2. 2 

ADS  /  LCS  Data  Telemetry 

LCS  shall  route  an  RF  transmission  cable  between  the  special  radio  and  the  TIS  buoy 
staging  area. 

3. 2. 2. 2 

ADS  /  LCS  Data  Telemetry 

All  RF  Cables  used  on  the  ADS  communications  link  shall  have  a  minimum  of  -12  dB  of 
shielding  effectiveness. 

3. 2. 2. 2 

ADS  /  LCS  Data  Telemetry 

LCS  shall  provide  120  /  208  VAC  60  Hz  3  Phase  and  120  VAC  60  Hz  single  phase  to  the  ADS 
radio. 

3. 2.2.2. 4 

ADS  Data  Reception 

The  LCS  shall  supply  1  transmit  antenna  and  a  receive  antenna  suite  with  360  degree 
coverage  and  adequate  instantaneous  bandwidth  for  ADS  TIS  telemetry. 

3. 2.2.2. 4 

ADS  Data  Reception 

The  LCS  shall  separate  the  ADS  TIS  receive  antenna  from  any  HF  transmitting  antenna  by  at 
least  1 00  feet. 

3. 2.2.2. 4 

ADS  Data  Reception 

LCS  shall  suppress  all  EMI  sources  in  accordance  with  MIL-STD-810. 

3.2.2. 3 

ADS  Shipboard  Radio 

Frequency  Antenna 
Characteristics 

LCS  shall  provide  shipboard  radio  frequency  antenna  with  the  following  properties: 

3.2.2. 3 

ADS  Shipboard  Radio 

Frequency  Antenna 
Characteristics 

LCS  shall  provide  a  continuous  metal  surface  from  the  antenna  mounting  location  to  the 
water.  This  will  provide  a  ground  plane. 

3.2.2. 3 

ADS  Shipboard  Radio 

Frequency  Antenna 
Characteristics 

The  lengths  of  the  LCS  routed  transmission  cables  between  the  ADS  transmit  and  receive 
antennas  and  the  special  radio  shall  be  less  than  150  feet. 

3. 2.2. 5.1 

Mutual  Interference 

The  LCS  HF  transmitter  shall  have  a  maximum  power  of  1  KW. 

3. 2.2. 5.1 

Mutual  Interference 

The  LCS  transmitters  shall  use  a  set  of  predetermined  frequencies  for  spectrum  overlapping 
the  ADS  operating  range. 

3. 2.2. 5.1 

Mutual  Interference 

The  LCS  shall  provide  ADS  with  a  list  of  planned  radiation  frequencies  overlapping  the  ADS 
operating  range. 

3.4.2. 1 

ADS  Hazard  of  Electro-Magnetic 
Radiation 

LCS  shall  locate  the  ADS  TIS  transmit  antenna  so  that  personnel  are  safe  from  radiation 
hazards  when  driven  by  a  500  Watt  PEP  transmitter. 
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Assumptions  Adjudication  Process 


Assumption 

No. 

Assumptions 

Where 
Should  Line 
Item  Be 
Documented 

ADS  Person 
(APM  &  SI  IPT) 
Resp  for  Insuring 
Implementation  & 

Rationale  and  / 
or  Clarification 

Tracking 

P.O.C.  on  ADS 

CCB 

Responsibility 

Change  to  ASW 
SPS,  LCS  ICD 
or  Other  Doc 

BCR  or  ECP 
Submitted 
(Y  /  N  &  Date) 

BCR 
or  ECP 
Status 

ASW  SPS,  LCS  ICD, 
or  Other  Document 
Requirement 

Associated  ADS 
Requirements 
Unique 

(Type  in  Doc) 

Unique  Identifier 

Identifiers 

ADS  SI  personnel  were  assigned  to  track  assumptions  from 
the  assumptions  table  to  ADS  and  LCS  documents  under  CM 

■  PMS  501  worked  collaboratively 

■  Status  was  reported  bi-weekly  at  SI  IPT  teleconferences  and 
quarterly  at  SI  IPT  meetings 
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6.0  Engineer-Engineer  Meetings 


■  LCS  Cost-type  contracts  did  not  provide  specific  hours  for  Sea  Frame  teams  to 
study  or  answer  government  questions  to  satisfy  GFI  to  others 


PMS  501  /  420  /  485  recognized  the  need  and  established  Engineer  to  Engineer 
Working  Groups  (Sea  Frame  /  ADS) 


■  PMS  485  developed  a  SOW  to  ensure  sea  frame  support  to  ADS  on  LCS 
integration  which  covered  COMMS  and  L&R  IPTs,  in  particular;  others  as 
needed 


PMS  501  conducted  Industry  Day  meetings  with  Sea  Frame  teams  to  highlight 
specific  technical  questions  and  to  identify  need  for  special  funded  studies 

■  Additional  SOWs  were  then  generated  to  address  Special  Studies 

■  Additional  resources  were  added  to  the  Gate  Keeper  meetings  to  monitor  & 
address  these  special  areas 


A  process  was  developed  to  provide  the  detailed  LCS  Sea  Frame  data 
needed  to  successfully  meet  ADS  CDR  design  requirements  and  SIT 
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Example:  ADS  Communications 

■  Lockheed  Martin  and  Harris  “RF  sub-contractor”  needed 
information  necessary  to  compute  RF  link  budgets  and 
to  design  to  its  KPPs 

•  6  simultaneous,  2MB  /  sec  direct  path  communication 
channels  from  a  buoy  to  ship  over  30NM  separated 

Impacted  LCS  and  ADS  power,  weight,  range,  antenna 
selection,  EMI  protection,  cable  shielding,  360  degree 
coverage,  etc. 
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Example:  LCS  /  ADS  Industry  Day 

Agenda 


Communications  (Segment  C) 

ADS  on  LCS  Communications  System  Overview 

Antenna  related  topics 

Spectrum  topics 

Decision  Brief 

Noise  topics 

Cabling  &  Interfaces 

Background  Brief 

•  Radio  Room  related  topics 

•  Summarized  findings 

•  Identify  Future  Actions  /  Special  Study  Requirements 

•  Determine  Schedule  need  for  future  meetings 

•  Provide  Meeting  Minutes  /  Actions  to  Gate  Keeper 


DISTRIBUTION  STATEMENT  C:  Distribution  authorized  to  U.S.  Government  agencies 
and  their  contractors;  other  requests  for  this  document  will  be  referred  to  CNO. 


UNCLASSIFIED 


30 


UNCLASSIFIED 


J * 


^  Example:  Buoy  to  LCS  Link  Budget 


Buoy  to  LCS  (6  ft  monopole  with  54  inch  Dixie  Cup) 


Frequency 


Transmitter  Power _ 

Antenna  Gain,  nominal  sea  state 


Cable  Losses  (Ant-PA,  T/R  switch 


EIRP _ 

Signal  Bandwidth 


Noise  in  Signal  Bandwidth _ 

Quasi-Minimum  Noise,  QMN 
factor  over  QMN 


Total  background  noise  level _ 

LCS  Front  End  Losses  (2:1  divider) 


Antenna  Tuning  Unit 


cable  loss 
Preselector  Loss 


LNA  Noise  Figure _ 

Receiver  Noise  Figure _ 

System  Noise  Figure 


Noise  Power 

Rx  Antenna  Gain,  14  ft  shipboard  monopole 


Path  Loss,  40  N  Miles 


Path  Loss,  30  N  Miles _ 

Receive  Power  (30  N  Miles) 


System  Design  Margin 


Operating  Range 

Es/No 


Reguired  Es/No _ 

margin,  30  Nmiles  -  sea  state  1,  no  ice _ 

margin,  40  Nmiles  -  sea  state  1,  no  ice 


6  Inches  of  Salt  Ice _ 

Buoy  Tilt  of  20  degrees  -  sea  state  4 


margin,  30  Nmiles  -  sea  state  4,  6  inches  salt  ice 


margin,  40  Nmiles  -  sea  state  4,  6  inches  salt  ice 


25.0 

30.0 

35.0 

40.0 

50.0 

50.0 

50.0 

50.0 

2.9 

2.9 

2.9 

2.7 

1.0 

1.0 

1.0 

1.0 

48.9 

48.9 

48.9 

48.7 

500.0 

500.0 

500.0 

500.0 

-117.0 

-117.0 

-117.0 

-117.0 

22.1 

20.0 

18.1 

16.6 

6.0 

6.0 

6.0 

6.0 

28.1 

26.0 

24.1 

22.6 

5.0 

5.0 

5.0 

5.0 

2.0 

2.0 

2.0 

2.0 

1.0 

1.0 

1.0 

1.0 

10.0 

10.0 

10.0 

10.0 

2.0 

2.0 

2.0 

2.0 

20.0 

20.0 

20.0 

20.0 

28.8 

26.9 

25.5 

24.5 

-88.3 

-90.1 

-91.5 

-92.5 

2.7 

2.6 

2.4 

2.1 

115.0 

119.0 

122.0 

124.0 
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Link  budget  factors 
driven  by  LCS 


Would  there  be  enough  gain  to  close  link  at  range  at  the  2MB  data  rate? 

UNCLASSIFIED 
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Example:  24/7  Operations 

■  [IRD20515]  LCS  shall  remain  within  direct  path  signal  range 

[SPS10448]  Direct  path  signal  range  shall  be  30  NMI  (threshold), 

45  NMI  (objective) 

■  [IRD20534]  LCS  shall  enable  communications  for  Sea  State 
4  (threshold),  6  (objective) 

■  [IRD20536]  LCS  shall  provide  acceptable  ADS  antenna  placement(s)  for  any 
azimuth 

■  [IRD20538]  LCS  shall  enable  communications  for  any  heading  change 

■  [IRD22538]  LCS  shall  enable  24/7  communications  for  [classified] 

■  [IRD20968]  LCS  equipment  supporting  ADS  activities  shall  be  designed  to 
prevent  an  ADS  Operational  Mission  Failure  from  occurring  due  to  loss  of 
data  reception  for  greater  than  15  minutes 

Would  the  LCS  design  support  the 
ADS  continuous  operation  requirement? 
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Example:  General  Antenna 
Requirements 


General  LCS-ADS  Antenna  Placement  Requirements 

IRD22537  BCR0026  #3 

LCS  shall  supply  1  transmit  antenna  and  a  receive  antenna 
suite  with  360-degree  coverage  and  adequate 
instantaneous  bandwidth  for  ADS  TIS  Telemetry 

IRD20518  BCR0026  #7 

LCS  shall  provide  a  continuous  metal  surface  from  the 
antenna  mounting  location  to  the  water 

IRD20891  BCR0040  #1 

The  LCS  sea  frame  shall  provide  the  ADS  shipboard 
antennas  with  protection  from  a  direct  lightning  strike, 
in  accordance  with  MIL-STD-464A,  paragraph  5.4 

IRD20922  BCR0026  #9 

LCS  shall  locate  the  ADS  TIS  transmit  antenna  so  that 
personnel  are  safe  from  radiation  hazards  when  driven 
by  a  500  Watt  PEP  transmitter 
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Example:  Transmit  Antenna  Details 


Shipboard  Transmit  Antenna  Properties 

IRD20519  BCR0038  #1  LCS  shall  provide  a  shipboard  transmit  radio  frequency 

antenna  with  the  following  properties: 

Operating  Frequency:  20  to  21  MHz  and  29  to  30  MHz 

Antenna  Gain:  >  3.0  dBi 

Antenna  Tuning  Unit  (ATU)  Loss:  <  1.0  dB 

Cable  Loss  (HPA  to  ATU):  <  1.0  dB 

Instantaneous  Bandwidth:  600  kHz 

■  VSWR<  2.0:1 

Polarization:  Vertical 

Azimuth  Coverage:  360  degrees 

Elevation  Beamwidth:  >  +  /  -  20  degrees 

ATU  Input  Impedance:  50  Ohms 

Power  Capacity:  >  500  Watts 
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Example:  Receive  Antenna  Details 


Shipboard  Receive  Antenna  Properties 

IRD20519  BCR0038  #2  LCS  shall  provide  a  minimum  of  2  shipboard  receive  radio 

frequency  antennas  with  the  following  properties: 
Operating  Frequency:  20  to  40  MHz 
Antenna  Gain:  >  2.5  dBi 
4:1  Divider  Loss:  <  6.5  dB 
Cable  Loss  (to  Receiver):  <  1.0  dB 
Instantaneous  Bandwidth:  20  MHz 
VSWR  <  3.0:1  at  output  of  antenna 
Polarization:  Vertical 
Azimuth  Coverage:  360  degrees 
Elevation  Beamwidth:  >  +/-  20  degrees 
Divider  Output  Impedance:  50  Ohms 
Antenna  Tuning  Unit  Losses: 

<  1.5  dB  @  40  MHz 

<  4.4  dB  @  20  MHz 

<  2.2  dB  @  25  MHz 

<  1.5  dB  @  30  MHz 

<  1.5  dB  @  35  MHz 

<  1.5  dB  @  40  MHz 
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Example:  Spectrum  Requirements 


LCS  Shipboard  Spectrum  Requirements 

IRD20531  BCR0026  #2 

The  LCS  HF  transmitter  shall  have  a  maximum  power  of  1  KW 

IRD20531  BCR0026  #1 

LCS  HF  Link  shall  use  a  set  of  predetermined  frequencies  [per 
communications  operational  plan] 

Requirements  on  ship  Transmitters  for  ADS 
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Example:  Noise  Environment 


LCS  Shipboard  Noise  Environment 

IRD20889  BCR0034  #5 

During  ADS  operation,  the  LCS  equipment  shall  not  generate  an 

E  field  greater  than  6  volts  /  meter  over  the  frequency  range  of 

17.5  MHz  to  45  MHz  at  the  ADS  receiver  antenna  locations 

IRD22535  BCR0026  #11 

All  RF  Cables  used  on  the  ADS  communications  link  shall  have 
a  minimum  of  -120  dB  shielding  effectiveness 
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Example:  Cabling  &  Interfaces 


Other  Requirements 

IRD22535  BCR0026  #12 

LCS  shall  route  a  RF  transmission  cable  between  the 
special  radio  and  the  TIS  buoy  staging  area  [on  board  the 
LCS] 

IRD22535  BCR0026  #4 

LCS  shall  provide  4  ea  chassis  locations  (3  radio  chassis 
and  1  controller  chassis)  in  the  radio  room  for  the  ADS 
radio.  Each  radio  chassis  will  conform  to  ANSI  /  EIA-310 
dimensions,  19  inches  wide  by  25  inches  deep  by  9U  high 

IRD22535  BCR0026  #5 

LCS  shall  provide  120  /  208  VAC  60  Hz  3  Phase  and 

120  VAC  60  Hz  single  phase  to  the  ADS  radio 

IRD20518  BCR0026  #8 

The  lengths  of  the  LCS  routed  transmission  cables 
between  the  ADS  transmit  and  receive  antennas  and 
the  special  radio  shall  be  less  than  150  feet 
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Example:  Size,  Weight  and  Power 

Estimates 


■  Space 

■  Qty  2<1>:  19”  EIA  31 OD  Racks 

•  182cm  height  [excluding  isolation] 

•  61cm  depth  [excluding  handles  &  connectors] 

■  Weight 

■  Less  than  1500  lbs  (1) 

■  Power 

■  <  1500  Watts  average  (1) 

■  <  3000  Watts  in  60  msec  peaks  at  3  sec  (typ)  intervals 


d)  per  LCS  ICD  Appendix  C  Section  C.2.6.4 
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7.0  Special  Studies 

Notwithstanding  SE-SE  efforts,  the  parties  could  not 
deliver  the  design  data  needed  to  develop  the  ADS  radio 
without  in-depth  knowledge  of  both  ship  designs  and 
electromagnetic  characteristics 

In  early  06,  Task  Instructions  were  issued  to  both  teams  to 
conduct  special  studies  using  a  Working  Group  structure 
administered  by  Gate  Keeper 

■  Both  teams  studied  Communications,  Launch  and 
Recovery  issues  resulting  in  sufficient  information  to 
successfully  meet  CDR  design  of  the  ADS  radio  and  L  &  R 
subsystem  with  no  claims  for  late  GFI  or  major  ECPs 

■  One  ECP  to  LCS  was  issued  to  provide  the  critical  Mission 
Module  support  changes  in  LCS  2 
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LM  or  GD  LCS 
Seaframe 
Manage  /  Design 
Team 


PEO(S)  (PMS-501) 
MSSIT 


ADS-on-LCS 
Web-enabled  IDE 


^ Gate  Keeper^ 
Web-enabled  IDE 
(Competition 
Sensitive 
Information)^ 


PEO  LMW  (PMS-420)  ASW  LCS 
Mission  Package  Lead 


PEO  LMW  (PMS-485)  APMs 


LM  ADS  Manage  /  Design  Team 


Sub-System  Design 
IPT  Leads 


Sub-System  Design 
IPTs 


Special  Information  Manager  (SIM) 

Ensures  only  PDF  docs  get  to  ADS 
contractors  &  monitors  ADS-to-LCS 
information  flow 
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8.0  Integrated  Master  Schedule  (IMS) 


Immm 


ID 


Task  Name 


Q2  |[  Q3  Jq4 


2006 


Q1  Q2  Q3  Q4 


2007 

Q1  I  Q2  I  Q3  )  Q4 


2008 


Q1  j  Q2  |  Q3  I  Q4 


2009 


Q1  Q2  Q3  Q4 


2010 


Q1  Q2  Q3  Q4 


Common  Milestones 

Seaframe  Common  Milestones  (PMS  501) 

Ship  2  (GD-1) 

Final  Design  Complete 
Ship  Check  Availability 
Ship  2  Delivery 
PDT&T  (Ship  2) 

Land  Based  CMS  SIM/STIM  Prod 
ASW  MP  Common  Milestones  (PMS  420) 

INCO  Milestones  -  TSE1HCO  TDA 

LCS  ASW  MP1  Certification  (NSWC-PC)  -  T&E/INt 
LCS  ASW  MP1  Installed  in  Seaframe  -  T&E/1NCO 
ADS  Common  Milestones  (PMS  485) 

Systems  Design  Development  Phase  (CLIN  3) 

CDR 

Post  CDR  Ship  Checks 

ADS  MM  SW  Package  Delivery  (ICP/ALP) 

TECHEVAL 

OPEVAL 


Final  Design  Complete 


Ship 


Land  Based  CMS  SIM  STIM  Prod 


IHCO  Milestones  -  T&E.1NC 


O  TDA 


LCS  ASW  MP1  Certificat  on  (H 


lase  (CLIN  3)  ^ 


Chech  Availability 

liip  2  Delivery 


tr 


PDTST  (Ship  ?> 


Post  CDR  Ship  Che - 


ISj^C-l 


PC)  -  TSE1HCO  TDA 
^  LCSASWMF1  Installed  in  Seaframe  -  TSE  Mi¬ 


ch  s 

ADS  MM  SW  Pack 


age  Delivery  (ICP/ALP) 


TECHEV4 

OPEVAL 
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Lessons  Learned 

■  Traditional  NAVSEA  and  ship  SE  processes  did  not  support 
concurrent  SE  of  2  interdependent  ACAT-1  programs 
reporting  to  2  sponsors  and  PEOs 

•  Outside  of  the  traditional  SYSCOM  business  practice 

•  Will  require  more  robust  inter-related  SE  and  T  &  E  processes 

■  Parties  must  mutually  respect  the  rigors  imposed  by  5000.1 
on  each  other 

■  Parties  must  agree  in  writing  on  capstone  requirements  and 
processes  to  be  implemented 

■  Metric  based  CM  must  be  imposed  on  critical  docs,  drawings 
and  processes 
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Recommendation 

ASN  (CHENG)  may  wish  to  include  a  section  on 
concurrency  in  the  SE  Guide  Book 

•  Lessons  Learned  and  the  ADS  on  LCS  8-Step 
process  offers  a  tested  process: 

-  The  use  of  a  IRD  to  capture  inputs  or  changes  to 
the  Interface  Control  Matrix 

-  Gate  Keeper  or  similar  management  structure 

-Assumption  documentation,  adjudication  and 
allocation  to  key  documents 

-  CM  of  Interfaces,  documents  and  processes 
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Systems  Engineering  Conference 


C-17  Systems  Engineering  Process 
to  Prioritize 

Material  Improvement  Program  (MIP)  Projects 


Major  Lardner 
516  AESG 


DSN:  986-9320,  Commercial  (937)  656-9320 
christopher.lardner@wpafb.af.mil 


Tom  Condron 
516  AESG 

DSN  986-4314,  Commercial  (937)  656-4314 
thomas.condron@wpafb.af.mil 
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Outline 


IS’  -*1* 

*  Purpose  of  Briefing 

*  Background 

*  Provide  User  with  recommendation  on  how  to  spend 
their  Material  Improvement  Program  (MIP)  money 

-  Starting  point  only,  user  makes  final  decision 

-  Proved  to  be  difficult 

*  Approach  used  to  provide  recommended  MIP 
Priority 
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Purpose  of  Briefing 
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*  Outline  the  approach  the  C-17  is  using  to  prioritize 
Material  Improvement  Projects  (MIPs)  for  funding 
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Definitions 


4f  «* 

*  Material  Improvement  Project  (MIP)  investigations 
are  initiated  when  a  Deficiency  Report  (DR)  is 
determined  to  warrant  further  investigation. 

*  MIPs  are  planned  engineering  investigations  to  find 
root  cause  and  corrective  action  or  evaluate 
proposed  enhancements. 
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Background 


Hi. 


»W» 


Deficiency  Reports  continue  to 
rise 

-  1800+  in  FY07 

-  2000+  projected  for  FY08 

About  250  -  275  Material 
Improvement  Projects  initiated 
annually 

DRs  &  MIPs  expected  to  rise  as 
more  aircraft  are  produced  and 
as  the  jet  gets  older 


MIP  Investigations  Initiated 
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MIP  Funding 
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■  On  Contract 


H  Projections 
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Background 


Hi. 


C-17  Supportability  Model  previously  used  to  for  initial 
prioritization 

Did  not  work  well  -  Highest  priority  projects  often  ranked  in 
middle  of  model 

Specific  concerns: 

»  Safety  -  Considers  only  the  initial  safety  RHI 
»  R&M  --  No  “bang  for  the  buck”  comparison  ability 
»  Subjectivity  in  many  fields 
»  Does  not  consider  MMH  or  RoR  Cost 
»  Subjective  factors  used  to  evaluate  operational  impact 
»  Attempted  to  assign  values  to  each  field 


Jill 


Provide  C-17  Systems  Group 
Recommendation  to  HQ  AMC 


Hi 


»w» 


Relatively  easy  to  prioritize  the  top  few  projects 

Very  hard  to  distinguish  between  lower  than  “top”  priority,  but 
still  good  projects 

-  Particularly  true  when  comparing  fundamentally  different  systems 
(e.g.  hydraulic  pump  reliability  versus  battery  charger  algorithm 
change) 

-  Tried  and  failed  to  reach  consensus  with  team  looking  at  narrative 
description  of  change 

Clearly  we  needed  a  tool  to  help  develop  a  priority 
recommendation 


Jill 


MIP  Prioritization  Tool 


*  « 
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Two  most  common  types  of  MIPs  are  R&M  and 
Safety  improvements 

-  All  R&M  projects  can  be  converted  into  dollar  savings 

»  Those  dollar  savings  can  be  converted  into  payback  period 

-  Safety  projects  can  be  quantified  in  terms  of  reduced  Real 
Hazard  Index  (RHI) 


MIP  Prioritization  Tool 
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•  Other  reasons  exist  for  MIPs  also 

-  Eliminate  dropped  objects 

-  Special  interest  item  (Flight  Crew  or  Maintenance  working  groups) 

»  Often  these  items  are  included  in  safety  or  R&M  concerns 

-  Over  and  Above  funding  drivers  during  depot  maintenance 

-  Pilot  workload 

-  Survivability  /  vulnerability 

-  Other 

•  Difficult  to  convert  these  considerations  into  common  terms 
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MIP  Prioritization  Tool 
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•  Nomenclature:  Deficiency  /  MIP  Payback  Model 

(  ) 

•  Excel  Spreadsheet  tool 

-  Initial  DMPM  for  new  Deficiency  Reports  (DRs) 

»  “Is  this  DR  worth  investigating?” 

-  Final  DMPM  for  MIPs  “Open  Awaiting  Funding” 

»  Calculates/summarizes  key  prioritization  information 

»  Provides  a  “bang  for  the  buck”  estimate  in  terms  of  simple 
Payback  Period  calculation 


11 


Top  Section  of 
Initial  Evaluation  DMPM 


m Li- 


A 

E 

C  |  □  |  E  II  F  |  G  1  H  |  1  |  J  1  K  |  L  |  0 

p 

q  nr 

s 

1 

Boeing  Proprietary  -  When  populated  with  Data 

2 

3 

Initial  DR  Response 

Revision: 

0.9 

4 

5 

DR  Title: 

Date: 

1  1 

e 

7 

DR  Number: 

1  1 

S 

Name  Phone  Number 

0 

COG  Engineer 

Pait  Number: 

i  H 

10 

SG  Engineer 

11 

Safely  Engineer 

WUC: 

i  H 

12 

R&M  Engineer 

13 

1 

14 

Is  there  already  an  existing  investigation  project  in  progress  for  this  Failure  Mode? 

YES 

|  Combine  With  DR=  | 

15 

NO 

Continue  with  this  Worksheet 

16 

17 

Summary  information  at  top  of  DMPM  tool 
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High  Impact  Issues  Section 
of  Initial  Evaluation  DMPM 


M Li. 


17 

18 

18 

1 

High  impact  issues 

Issue 

20 

SAFETY 

* 

If  the  Safety  Issue  =  "y",  recommend  SG  Status  of  "09  -  Combine"  or  "03  -  MIP  Investigation 

21 

If  Mai  Safety  Issue  is  "y",  then  Safety  will  automatically  conduct  an  RHI  Analysis 

22 

CORT 

The  RHI  value  will  be  entered  on  the  next  page  "Final  MIP  Response" 

23 

24 

SORT 

* 

25 

26 

Other 

* 

27 

28 

29 

PICR  Humber 

PAT  Rmih 

30 

PICRAC 

noN 

TEAM  (PAT) 

31 

If  the  answer  is  “Yes”  to  any  of  these  questions  -  The  Deficiency  Report  is 
investigated 

•  Other  includes  factors  such  as  “Dropped  Object”  and  “Pilot  Workload” 
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Reliability  Impacts  Section 
of  Initial  Evaluation  DMPM 


34 

2 

Re//aM/ry  /mpacrs 

Actual 

Estimated 

%  Deviation 

Potential 

Improvement 

35 

All  RMfcA  data  based  on  Ship  Set 

38 

NMC  Hours 

Hrs 

$0 

37 

33 

PMC  Hours 

$0 

39 

40 

MMH  Hours 

Hrs 

$0 

41 

42 

MTBR  Hours 

1  \ 

Hrs 

Hrs  | 

|  0.0% 

Hrs 

Deviation  is : 

43 

L / 

rn 

cn 

44 

Fleet  Flight  Hours 

Yel  =  -10%  to  10^ 

45 

■ 

F - i 

Red  <  -IQtf 

49 

Quantity  Per  Aircraft 

Units 

47 

RMfcA  data  based  on  the  last  tvo  fears  of  Flight  Hours 

49 

Converts  Reliability  Impacts  into  cost  numbers 

•  Assumes  solution  will  completely  eliminate  source  of  Not  Mission  Capable 
(NMC),  Partially  Mission  Capable  (PMC),  and  Maintenance  Man  Hours  (MMH) 

•  Not  Mission  Capable  Hours  =  $2000  each 

•  Partially  Mission  Capable  Hours  =  $1000  each 

•  Maintenance  Man  Hour  =  $72  each 
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Replacement  /  Repair  Cost  Section 
of  Initial  Evaluation  DMPM 


A  |  E  |  C  1  0  1  E 

F 

G 
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1  |  J  |  K  1  L  |  0 
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Q 

R 

49 

50 

3 

Replaceme 

nt/ Repair  Cost 

51 

Repaii  able 
Unit 

52 

My"=  VESand  MnM=  MO 

53 

Unit  Cost  (FEDLOG  GOLD) 

i  i 

54 

55 

Replacement  Cost  /  Year 

| 

1  sol 

1 

1 

1  W 

56 

57 

Estimated  Repair  Cost  /  Year 

1 

1  sol 

1 

1 

i  $0 

56 

59 

Other  Savings/Avoidance  Year 

1 

1  1 

1 

1 

1  so 

60 

61 

Explain  Source  of  Savings.  Avoidance 

| 

1 

62 

63 

64 

65 

r-r- 

Calculates  the  Repair  /  Replacement  cost 

•  (Failures  per  FH)  *  (QPA)  *  (FH  in  a  year)  *  (cost  per  repair/replacement) 
Other  Savings  -  e.g.  fuel  savings  for  a  nacelle  sealing  improvement 


Summary  /  Total  Section 
of  Initial  Evaluation  DMPM 


A 

B 

c 

□  E 

F 

G 

H 

1 

J 

K 

L 

0 

p 

Q 

R 

67 

4 

68 

Summary  Totals 

68 

70 

Potential  Savings  /  Year 

1  io| 

1 

71 

72 

Replacement  .  Repair  Costs 

1 

1  551 

P 

— 

73 

74 

T otai  Potential  Savings  i  Year 

1  5o| 

P 

75 

76 

77 

78 

MIP  Investigation  Recommended?  | 

78 

80 

Instructions: 

81 

. 

82 

Section  1  (Lavender  color)  completed  by  Safei 

ly  Engineering 

Consider  the  following  Alternative  So 

utions  or  Others  Not 

Liste< 

83 

Section  2  (Yellow  color)  completed  by  AV1ET 

a.  Tech  Pub  Change  (PCR) 

84 

Section  3  (Green  color)  all  header  boxes  completed  AVIET 

b.  Technical  Order 

85 

c.  PICR 

86 

d.  RIPTA 

87 

The  Grey  boxes  are  calculations  and  do  not  get  filled  out. 

e.  Close  DR 

Calculates  the  Repair  /  Replacement  cost 

•  Total  Potential  Savings  /  Year  =  Sum  of  “Potential  Savings  /  Year”  +  “Repair  / 
Replacement  Costs” 

•  If  total  Potential  Savings  /  Year  exceed  $100K  -  Investigate  DR 

•  If  any  of  the  questions  in  the  Top  Section  of  the  DMPM  are  “YES”  -  Investigate  DR 
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yjrTTTJ 

/uuAAA 


Hi. 


4f 


»W» 


Top  Portion 

of  Final  Evaluation  DMPM 


c  D  E  F  0  H  I 


±1  K  LlI 


0  P  Q 


^_A 


2 

1  i 

3 

Initial  DR  Response 

Revision: 

0.9 

4 

5 

DR  Title: 

Date: 

6 

7 

DR  Number: 

8 

Name 

Phone  Number 

9 

COG  Engineer 

Pan  Number: 

10 

SG  Engineer 

11 

Safely  Engineer 

V 

VUC: 

12 

RSM  Engineer 

13 

14 

Is  there  already  nil  existing  investigation  project  in 

irogress  for  this  Failure  Mode; 

YES 

Combine  With  DR? 

15 

NO 

Continue  with  this  V 

/orksheet 

16 

Summary  information  at  top  of  DMPM  tool  -  Same  as  Initial  Evaluation 
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High  Impact  Issues  Portion 
of  Final  Evaluation  DMPM 


» 


IS 

10 

High  impact  Issues 

Issue 

Current  RHI 

Predicted 

RHI  After  Fix 

20 

SAFETY 

1 

r 

21 

22 

CORT 

| 

r 

23 

24 

SORT 

1 

r 

25 

20 

Other 

1 

[ 

27 

PICR  Humber 

PAT  Ranh 

20 

PICR  ACTION  TEAM  (PAT) 

1 

r 

29 

Only  difference  from  Initial  Evaluation  Tool  is  Addition  of  Current  Real 
Hazard  Index  (RHI)  and  Predicted  RHI  after  fix 
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Reliability  Impacts  Section 
of  Final  Evaluation  DMPM 


m 


30 

31 

2 

Re/iaM/iy  /mpacrs 

Actual 

Reduced 

Potential 

Improvement 

Value  of  Potential 

Improvement 

32 

All  FtMfrA  data  based  on  Ship  Set 

33 

NMC  H  ours 

Hrs 

0 

Hrs 

0 

Hrs 

$0 

34 

Percentage  Value  of  PMC 

35 

PMC  Hours 

Hrs 

0 

Hrs 

50% 

0 

$0 

30 

37 

MMH  Hours 

Hrs 

0 

Hrs 

0 

Hrs 

$0 

33 

Trend  Is: 

39 

Predicted 

%  Deviation 

40 

MTBR  Hours 

jJ 

Hrs  | 

0 

Hrs 

A 

Hrs 

RediStf 

41 

i  ti 

yim 

i  J 

|  Ed  | 

42 

Fleet  Fligl  it  Hours 

i^VBI 

wm 

43 

LI 

u 

IVjM 

| 

44 

Oikiinhy  Pei  Aircraft 

r  i 

Units"  *  M 

45 

FtMfrA  data  based  on  the  last  tvo  pears  of  Flight  Hours 

i  n i  i  i i  i i i i i 

Converts  Reliability  Impacts  into  cost  numbers 


•  Engineer  inputs  estimated  value  for  metrics  after  fix  is  incorporated 

•  For  PMC  hours,  engineer  also  estimates  percentage  value  of  a  PMC  hour 

•  For  minor  items  it  may  be  20  percent,  for  more  significant  items  it  may  be  75 
percent 


•  Change  from  predicted  value  of  MTBR  hours  is  also  calculated 
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Replacement  /  Repair  Cost  Section 
of  Final  Evaluation  DMPM 


^JTTTTJ 

fUnJtm Aan*n 


W 


»w» 


46 

47 

3 

Replacement  i  Repair  Cost 

46 

49 

Repairable 

Unit 

50 

51 

Unit  Cost  fFEDLOG  GOLD) 

,  1 

1 

1 

52 

53 

Replacement  Cost  /  Year 

1 

1  io| 

1  1 

1  jo| 

1 

so 

54 

55 

Repair  Cost  /  Year 

1 

1  so 

1 

1  so| 

so 

56 

57 

Other  Savings/Avoidance  .  Year 

sol 

1  1 

1  1 

1 

so 

59 

59 

Source  of 

60 

61 

Spares  {FEDLOG): 

Modify  Existing  Spar es 

1 

1  1  W 

62 

63 

r-  4 

Calculates  the  Repair  /  Replacement  cost  -  Very  similar  to  Initial  DMPM 
•  Based  on  predicted  level  of  improvement 


Summary  /  Total  Section 
of  Final  Evaluation  DMPM 


65 

4 

Summary  Totals 

66 

Value  of  Potential  Savings  .'Year 

1  Ml 

67 

68 

Potential  Improvement  Cost  (NRE) 

1 

68 

Potential  Improvement  Cost  (Recurring) 

70 

Potential  MIP  Total 

$0 

71 

Payback  Period  in  Years 

0.0 

72 

73 

Potential  Cost  for  Spares 

$0 

74 

Potential  MIP  Total  plus  Spares 

$0 

75 

Payback  Period  including  Spares 

0.00 

76 

77 

78 

Instructions: 

78 

80 

Section  1  (Lavender  color)  completed  by  Safety  Engineering 

81 

Section  2  (Yellow  color)  completed  l>y  AV1ET 

82 

Sections  3  &  4  (Green  color)  completed  l>y  Project  Manager  or  A  VIET 

83 

The  Grey  boxes  are  calculations  and  do  not  get  filled  out. 

•  Calculates  Cost  of  MIP  Implementation 

•  Calculates  savings  per  year 

•  Calculates  Payback  Period 

•  Both  with  and  without  spares  cost  (spares  are  not  paid  out  of  MIP  funds) 
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Jill 


Prioritization  Process 


'*  '  w" 

,JS»  ^5 

•  Group  MIPs  Open  Awaiting  Funding  into  four  categories 

-  Top  Priority  -  Fund  out  of  cycle  if  possible 

-  Good  Projects  if  funding  is  available  (in-cycle) 

-  More  Information  Needed  to  rank 

-  Close  as  acceptable  risk  -  We  prefer  to  close  these  as  early  as  possible 

•  Aircraft  Systems  IPT  (ASIPT)  rank  their  MIPs 

-  ASIPT  has  the  most  MIPs 

-  Each  MIP  has  summary  data  from  the  DMPM  at  the  top  along  with  a 
narrative  project  description 

•  IPT  Technical  Leads  meet  and  insert  other  MIPs  into  the  list 

-  A  meeting  is  held  with  lead  using  command  (HQ  AMC)  to  present  C-17 
System  Group  MIP  Priority  recommendation 
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DR/MIP  Process 
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4^ 


X32629  NEW  (05  DEC  2005) 


Input/Output 


Task 


Decision 


/"Go  toN  In-Process 
'^X.XXy  interface 


Connector 


©Kecor,  A0?!”' 


DR/MIP  Process 
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10th  Annual  NDIA  Science  &  Engineering  /  DoD  Tech  Expo 
“Reducing  Technology  Risk  in  Acquisition  Programs” 

Testing  Concept  of  Operations  (C0N0PS1 
In  DoD’s  Net-Centric  Environment 


Mr.  Steve  Reeder 

SCRA 

5300  International  Boulevard,  N.  Charleston,  SC  29410 

steve.reeder@isn-scra.orn 

IP)  757-203-4421,  (F)  043  700-3250 
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Celle  ha  ruling  Ta  Advance  Tedmelogy 


October  22-26,  2007 
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GCCS-J  4.x  External  Interface  Architecture 
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DoD  s  resgonsihility  is  the 
management  of  vlelence. 


Principles  if  War 


Regardless  the 
nf  Technology 


•  Objective 

Clearly  defined,  decisive  and  attainable 
objective.  Each  operation  must  contribute  to  the 
ultimate  strategic  aim. ... 

•  Offensive 

Seize,  retain,  &  exploit  the  common  objectives. 
Means  to  maintain  freedom  of  action  &  achieve 
decisive  results. 

•  Mass 

Synchronizing  all  the  elements  of  combat  power. 
Mass  the  effects  not  necessarily  the  forces. 

•  Economy  of  Force 

No  part  of  the  force  should  ever  be  left  without  a 
purpose 

•  Maneuver 

Movement  of  forces  in  relation  to  the  enemy  to 
gain  positional  advantage.  Continually  pose  new 
problems  for  the  enemy  by  rendering  his  actions 
ineffective  &  eventually  defeating  him. 


•  Unify  of  Command 

For  every  objective,  seek  unity  of  command  and  unity  of 
effort.  Unity  of  command  means  that  all  the  forces  are 
under  one  responsible  commander 

•  Security 

Never  permit  the  enemy  to  acquire  unexpected  advantage. 
Protecting  the  force  increases  friendly  combat  power.. 

•  Surprise 

Strike  the  enemy  at  a  time  or  place  or  in  a  manner  for 
which  he  is  unprepared 

•  Simplicity 

Prepare  clear,  uncomplicated  plans  and  concise  orders  to 
ensure  thorough  understanding  effectiveness 


J^SCRA 
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First  and  foremost 


USJFCOM  is  the 


Joint  Warfighter 

Advocate 


JOINT  CAPABLE  FORCES 


.7SCRA 
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FORCE  EMPLOYMENT 

Air,  Land,  and  Sea  Operations 
CAS  Planning  and  Execution 


FORCE  PROJECTION 

Joint  Operation  Planning  SITUATIONAL  AWARENESS 

&  Execution  System  (JOPES) 

ADAPTIVE  PLANNING 


SITUATIONAL 

AWARENESS 

Common  Operational 
Picture  (COP) 


FORCE  READINESS 

Readiness  Assessment  System  (RAS) 

Global  Status  of  Resources  and 
Training  System  (GSORTS) 


INTELLIGENCE 

Integrated  Imagery  Intel  (13) 


FORCE  PROTECTION 


Early  Warning  and  Integrated  Air 
and  Missile  Defense 


J^SCRA 
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Program  Decision  Memorandum  (PDM)  III, 
December  20,  2005,  Tasked  the  Assistant 
SecDef  for  Networks  &  Information 
Integration  /  DoD  Chief  Information  Officer 
(ASD(NII)  /  DoD  CIO.... 

“To  accelerate  the  provisioning  &  adoption 
of  Core  Enterprise  Services  (CES)  across 
DoD. 

In  commercial  industry  speak,  that  means  to 
start  developing  a  System  Oriented 
Architecture  (SOA)  approach  for  C2. 


"'SCR  A 


©2007  By  SCRA.  All  Rights  Reserved 


The  DoD  must  continue  to  evaluate/assess  technology’s 
impact  on  the  current  war.  And  quickly  adopt  approaches  that 
increase  our  combat  capabilities 


•  Emerging  technologies,  like  SOA  and  innovative 
CONOPS  must  accelerate,  together 

•  Viable  technologies  must  be  rapidly  integrated 
into  current  C2  practices,  allied  operations, 
training,  and  doctrine  for  maximum  effectiveness 

•  Warfighter  needs  are  dynamic,  our  coalition 
arrangements  are  unique,  and  the  “funding- 
requirement-acquisition”  process  is  unacceptable 
in  the  ‘immediate’  for  the  soldier  on  the  patrol 


We  believe  that  Net-Centric  Environment  “e.g.  SOA  approach”  is  the  next  principal 
mechanism  for  enhanced  Command  Capability  of  Joint  C2. 


J^SCRA 


©2007  By  SCRA.  All  Rights  Reserved 


urn 

_  > 

/ 

Familiar 

Less  familiar 

What  we  use 

FOCUS 

What  we  use  and  how  we  use  it 

Technology  affects  on  system 

SOLUTION 

Technology  +  method  +  people 

capability 

affect  on  operational  capability 

Developers'  perspective 

PERSPECTIVES 

Warfighter  perspective 

Hardware  and  software  must  be 

CENTRAL  RULE  or 

Materiel  and  non-materiel  must  be 

developed  together 

CONCEPT 

developed  together 

SoS  assessment 

APPROACH 

MCP  assessment 

-  OT&E  focus  on  the  system 

-  Holistic  focus  on  all  components 

System  centric 

CENTRICITY 

Capability  centric  (Warrior) 

Focus  on  Joint 
Warfighter’s  urgent 
operational  need  - 
solution  providers  must 
forge  a  single 
1 integrated ’  enterprise 
to  reduce  risk  in 
satisfaction  of  that 
need. 


Changing  the  Business  Model  Requires: 

(1)  Willingness  to  work  together  to  leverage  each  others  core  competencies 

(2)  Focus  on  Joint  Warfighter  as  central  driver  -  solution  need  originator  and  evaluator 

(3)  Commitment  to  providing  meaningful  services  rather  than  inflexible  “products” 


J^SCRA 
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■  TRANSFORMATION  EFFORTS: 

•  Moving  away  from  a  Soviet  based  system 

•  Moving  to  a  professional  as  apposed  to  a  conscript 
based  force 

•  Moving  to  a  capitalistic  based  economic  model 

•  Moving  to  asymmetric  warfare 

•  Moving  to  a  net-centric  combat  capable  force 


CliLOJV 


At  the  request  of  Poland’s  Chief  of  Defense  (CHOD),  a  combined 
NATO  and  USJFCOM,  Poland’s  Military  staff,  plus  Industry  and 
Academia  constructed  a  near  term  Common  Operating  Picture 
(COP). 

Constructed  a  near  term  SOA  environment  to  integrate  Poland’s  Air, 
Land  and  Sea  into  a  combined  Common  Operational  Picture. 

Supported  Poland’s  role  as  a  NATO  member  &  US  strategic  partner 
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BRITE  interface  incorporated  in  every  system 


□ 


Automatic  discovery  add-ons 


BRITE  =  Baseline  for  Rapid  Iterative 
Transformational  Experimentation 


CHOD& 
Operational 
Command’s 
Critical  Information 
Requirements 


J^SCRA 
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Integrate  Existing  and  Emerging 
C2  Capabilities 


Sustained  by  Global  Information  Grid  Enterprise 


Integrate  Soluti 
with  DoD’s 
Net-Centric 
Data  Strateg 


Support 

Enterprise-based 

Joint 

Architecture 


These  web  systems  and  services  will 
have  a  unique  combination  of 
characteristics  that  differentiate  them 
from  more  conventional  legacy  client 
server  applications.  In  particular,  they 
tend  to  include: 

•  Architecture  places  data  at  the 
center  of  its  design:  Enterprise 
Resource  Pattern  (ERP)  & 
Enterprise  Service  Bus  (ESB) 

•  ERP  standardizes  access  to 
any  C2  domain  object  (APIs) 

•  ESB  publishes  messages 
based  on  an  event/trigger 


Services  and  Net-Centric  Enterprise  Services  ...  .  .  .  . 

•  Rapidly  changing  technologies, 

e.g.  more  actors,  platforms, 

networks,  and  services  not 

applications 


j^SCRA 
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Current  C2  Systems 


Candidate  Increasing 

Services  Capability 

Maturity 


Net-Enabled  Command 
Capability 


Systems  transformed  into 
discreet  service  capabilities 


C2 

Capabilities 


Services  used  to 
support  Joint 
‘business’  processes 


Systems  Focus 
Stove-piped  systems 
Information  push 


“The  Sandbox” 


Operations  Focus 
Net  &  Data  Centric 
Information  Pull 


Joint  Warfighters  Command  and  Control  Need  Driven 


With  the  net  centric  approach,  user  engagement  occurs  in  the  “ sandbox ” 
during  the  combined  evaluation  referred  to  as  the  “piloting”  events. 


J'SCRA 
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■ 


REQUIREMENTS 


Capability 

Articulation 


System 

Engineering 


OPERATIONS 


Capability 

Validation 

and 

deployment 


Testing  & 
Evaluation 


ACQUISITIONS 


PERSISTENT  T&E  ENVIRONMENT 


J'SCRA 
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Industries  Mixed  Results 


Mixed  Results 

How  has  your  company's  SQA/Web 
services  adoption  performed? 


32% 
Fallen  short 
of  expectations 


0% 

Eapft 

expectations 


%  Met 
expectations 


Data:  Inlormation  Wuck  Research  SOA/Web  Sol-vltaS  survay 
ol  278  business  technology  professionals;  228  companies 
using  SOA/Wc-b  services 


J'SCRA 
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^  j - 1 9  PH 


David  Linthicum 


Linthicum  Group 

Top  5  Mistakes  w/  SOA,  wd< 

1.  Not  enough  trained  IT/SOA  architects  to  put  on  the  problem. 

2.  “Manage  by  Magazine”  approach  to  SOA. 

3.  Don’t  understand  the  unique  nature  of  their  problem  domains. 

4.  Treat  SOA  as  a  project,  not  a  journey. 

5.  Unable  to  define  the  value. 


“I  Ac lively  feck  Davidsaid. 

fOAStanZadT120me^ 


»n  time.” 


Jim  Green, 

Designing  Reusable  Software, 

-  Types  of  services: 

(1)  put  data  in,  (2)  get  data  out 

-  SOA  &  error  handling  =>  careful  planning 


IT’s  Challenge  —  Deliver  the  Data  with  Flexibility  &  Agility 


Intel 

Systems 


Data 

Services 

Layer 


Intel 

Data 


mm 

t\  .  1 

= 

FwH  — r  tjj'r-171 

—  —  tan 

-  s 

'  '  1 

Dashboards  Reporting  Applications 

4^4,4- 


“No  SOA  plan  is  complete  without 
a  data  services  /ayer" 

Source:  AMR  Research 


Constant 

Change 


Virtualizes 

a 

Abstracts 


Siloed 

& 

Complex 

COMPOSITE 
v\\  \  >1 1: 
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Hub  Vandervoort,  CTO,  Progress/Sonic 

His  Key  concept  was  Enterprise  Service  Bus  (ESB) 
Service  Requires  alignment  across  4  dimensions 
Functional,  (2)  Structural  (3)  Behavioral  (4)  Performance 

Interaction  Model  (-Request  Reply,  -Store  &  Forward,  - 
Pub/Sub,  -Bulk  transfers) 


Mechfwtooy 

COfttutfcaitt* 

Steve  Kahn,  Bearing  Point 

•  Discussed  two  SOA  projects  (Insurance 
Company  &  Commercial  packaging  firm) 

•  Focus  on  the  business...,  technology  is 
never  enough. 


Some  Rnal  Thoughts 


■  SOA  Maturity 

■  Incremental  approaches  work  best 

■  Expect  to  get  smarter  along  the  way 

■  Business  Process  Management  and  SOA 

■  BPM  is  the  ultimate  enabler  of  return  on  SOA  investment 

■  BPM  is  to  SOA  what  a  conductor  is  to  an  orchestra 

■  Business  processes  are  built  from  high-level  composite 
servi  ces 

■  Invoke  business  processes  as  services 

■  Knock  down  remaining  impediments 

■  Maintain  Leadership  Support 


SOA  Case  cs 


r.-adav  3c-..  - 


J?JT=ei^ATI<3!  ^  SQ^UTlfl-^  3«i 


J^SCRA 


©2007  By  SCRA.  All  Rights  Reserved 


16 


Booz  |  Allen  |  Hamilton 

delivering,  results  tn art  endure 

Melissa  Soley,  BAH,  Trans-National  COP 

■  BAH  Mission  Engineering  (ME)  method  is  a 
bottom-up  IER  data  capture  approach 

Very  intensive  data  capture  approach 

■  Point  of  interest:  80%  of  an  Intel  Analyst’s  time 
is  spent  simply  retrieving  data  not  analyzing. 


High  Level  Operational  Architecture 


OPERATIONAL/STRATEGIC 


QUADRANT 

■  Int ell  Aichitectuie  Integi.ition 

■  System  Development  Compliancy 

■  Piogi.im  of  Record  Alignment 

■  Ability  to  Standardize Impact  DCR 

Materiel  Solutions 


TACTICAL/OPERATIONAL 


QUADRANT 


■  DCGS  JCD  COII OPS  Development 
-  COALITION  IPT  Support 
■  DDTE  Implementation  Benefits 
■  DCR  Development  Harmony 
■  JIOC  Mi t nation  Process 


J'SCRA 
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AmberPoint 

Sean  Fitts,  Amber  Point 

•  Keys  to  SOA  Runtime  Gov’n 

Visibility  =>  what  is  going  on  &  whols 
using  it? 


Control  =>  Actions  to  prevent  or 
correct  issues 


Integrity  =>  Ensuring  changes  don’t 
impact  the  whole  infrastructure 


The  SOA  Validation  Problem 

*  Business  System  Integrity  Always  at  Risk 


Service  reuse  creates  dependencies 

Impact  of  any  changes  ripple  throughout  the  system 

Real  impact  of  planned  changes  is  hard  to  predict 

Impact  of  unplanned  or  unannounced  changes  can  be  devastating 

Yet,  it  quickly  becomes  impossible  to  setup  and  replicate  all  dependent 
systems  fior  testing  elsewhere 

Need  way  to  continuously  check  for  integrity  -  both 
in  staging  and  in  production 

Amber  Poimt  0.uit  14 
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Universal 

Joint 

Task 

Lists 

(UJTLs)  ^ 


.Governance 


Test 

Threads 


lanagement 


Master 

Training 

Guide 

(MTG) 


iecurity 


service 

Discovery 

Mediation 


Other  ^ 
Authoritative 
Sources 


Reporting 


Messaging 


Operational 

Community 


Operational  Requirements 

Subject  Matter  Experts  (SME) 


Materiel  Developer 

Subject  Matter  Experts  (SME) 


System 
Community 


So  what  did  he  sayP 


DoD’s  C2  environment  has 
@  7  million  customers 

Our  business  is  the 
management  of  violence 

JFCOM  is  the  Joint 
Warfighter  Advocate 

DoD  is  moving  to  Net 
Centric  C2 

DoD  will  continue  to  adapt 
to  change 


Poland’s  military 
transformation  & 
movement  toward  Net 
Centricity 

NECC  programmatic 
processes 

Industries  views 

NECC  testing  concept 

DoD  is  in  the  early 
stages  of  SOA  adoption 


J^SCRA 
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BACKUP  SLIDES 


Use  an  Operational  Mission  Thread  Concept 


SHARED  SITUATIONAL  AWARENESS 


Operational  Event  Trace 
Description  (OV-6c) 


Event  Table 


J'SCRA 
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Systems  min  King 


The  Art  of  Seeing 

Robert  J.  Monson,  Ph.D. 

Lockheed  Martin  Tactical  Systems 


S  u  mm  a  ry 


ng  and  Projects 

Ssii rts^T  Co n cr^WrAbs  tract  Thinking 
Seeing  Systems 
Whole-Brain  Thinking 


r T, 


HheJDevelop 


especially: 

-  The  human  body  regarded  as  a  functional  physiological  unit. 

-  An  organism  as  a  whole,  especially  with  regard  to  its  vital  processes  or  functions. 

-  A  group  of  physiologically  or  anatomically  complementary  organs  or  parts:  . //?e 
nervous  system;  the  skeletal  system. 

-  A  group  of  interacting  mechanical  or  electrical  components** 

-  A  network  of  structures  and  channels,  as  for  comnauiii^lj^J^y^ror  distribution: 

-  A  network  of  related  computer  software,  hardware,  and  aatanransmission  devices. 

An  mgwmed  set  of  interrelated,  ideas  or  principles.. 


rssflTcatTon  orana 


economic,  or  political  organizational  form. 

'ally  occurring  group  of  objects  or  pher 
of  obi 

'  ius,  ordei 

rganized  and  coordinated  method;  a  procedure. 

The  prevailing  social  order;  the  establishmenflUsed  win  the:  You  canTti&aF 
the  systems 


(Sherwood,  2002) 


J 


jUU  wish  to -understand  a  system,  and  so  be  in  a 
position  to  predict  its  behavior, 

Cutting  it  up! into  bits 

For  study  is  likely  to  destroy  the  system’s 
connectedness,  and  hence  the  _ 

If  you  wish  to  influence  oi^controwWbShciVior  of  a 
svstem. 


She  place  in4 


i&m 


¥>i-ll  :rfc.t*}UZf\  r r r  anarhier.  ifiS-ciQ 


SdTTess  is  alfaootift' 


ngineering 


Related  to  Problem  Solving 

— ^-eonn^cfiwg  ordispaSe  ideas  / concepts 
■  Relating  previoysly  unconnected"  elements 


Like  Project  Management  on  steroids 

J  xJ 

-  Understand  whafyou  j^Jgnd  tojdo 

rk  required 


-  ^xmum  (na.r-jlg.ici 


^SffTas  yocrgo  forward 


tralrare  Trie  success  and  fall  are 


modes  of  typical!  endeavors? 


(Blandish  Group  CHAOS  results^ '1 998) 


i-up 


r  actors 


roivement 


-  executive  management  support 

-  clear  requirements 

-  proper  planning 

-  realistic  expectations 

-  smaller  project  milestones 


’Worfmg,  focused  staff 


Failun 


Top 

-  lack  of  user  input 

-  incomplete  requirements 

-  changing  redpte'ments 
H  lack  of  executive ^upport 

-  technology -ineompetence 

-  lack  of  resources 
unrealistic  expectations 


iimmr 


-  urw 


ealistic  timeframes 


-  new  technology 


(Belassi  &  Tukel,  1996) 

Top  management  support  throughout 

■  Proper  goal  definition  and  setting 

■  Accurate  estimates 
End-user  involvement 

ftProject  manager  on  site 
^Appropriate  team  selected 

■  Testing  and  training 


-  no  historical  software  measurement  data 
^^BfaiFure  to  use  automated  estimation  tools 

-  Tailure®to  use  automated  plannmg  tools 

-  failure  to  monitor  progress  qjjjmilesteraes^ 

-  failure  to  use  effective  architecture^ 


— -  -  failure  to  use  effective  developmen 


Tl 


Formal  configuration  management 
Hmore  tfialf  30%  creep  in  requirements 


~i0i  :: 


to 


project  fsiikiree 


SEEI 


J 


oor  requirements  definition 
Inadequate' software  process  ffianagemarif! 


■  Lack  of  integrated  product  teams 
9  Ineffective  s u b co n tra cj rpa na g erne nt 


■  Too  little  attention  to. softwaPe-sprcbiteolRj re 


j 


iiiusttp]!  liw 


standards 


'sanware'Sy stern a  railureMods 

(Website:  nitpV/sursijJurn^sdu^k) 

-Hostile  £u\m 


i ) 


fs- 


uui  reporting  structure 

-  Sponsors  are  not  involved 

Whe  Success  of  Any  Project  is  Lffked  to  tie  People 

-  Take  care  of  the  people  to  succeed 
Over-commitment 

-  Unclear  expectations  and  understanding  of  the  smpa 
Decision” Making  Escalation 

-  Movement  becomes  more  important-thafnDTSjTess 
Pre 

idden  Ag 


J  ~  i  rr  r  rrj  rrfH  r"/-- r  n  cm  La  b 


oring  ine  Human 
^ie  lure  of  the  leading  edge 
-  It  can  be  costly  and  heartbreaking 


J 


(Website:  htipV/surcijJurns^edu.pk) 


illfiQ 


j 


d  by  individuals  who  are  not  informed 

aBadeamte  consSlfaS^^w&hSS^jof  stake  hollers 


IV/V/VJU 


in  the  dark 


Design  by  committee 

-  No  major  responsibility  holder 

technical  fix  for  a  management  problem 

-  Lack  of  whole  picture  thinking 

Staff  turnover 

-  Lack  of  engagement 


Competency 


e  by'the  wrong  people 


o^rcJiinai 


estones 


equate  testing  and  training 

-  Not  having  discipline  in  our  development 

Communicatora  problems  and  work  group  problems 

-  As  ^always... 


I  r(  | — \  J  i  wF| 

Leri  Brain  D'iyJ 


P 


s  io  veroai  insirucnons 
^■©telem-eelvee-log  ica  l  ly-  afld  seqtrefl'tiaily 
ooks  at  differences 
Is  planned  and  structured 
Prefers  established,  expected  information 
Prefers  talking  and  writing 


J  . 
i 


Prefers  multiple  choice  tests 


_ _ i_ _ i_  r _ IE _ 


'•JPJ  rfO  J^T^I  ff. 


gs 


BEUhSBRbR  structun 


Cl  5Wf. 


->  Js  s 

-I 


SHU 


r  Cl  0|vJI  itter,  distinction  is  important 
Draws  off  previously  accumulated;  organized  information 


Right  Brain  Style 


ir-ebiem-sol  V'©&-wra^5yfl€ResPooKS  TonfattSTs 
0(5K§  cit  51TflliajEITlSs 
Is  fluid  and  spontaneous 
Prefers  elusive,  uncertaiH  information 

Prefers  drawing  and  maflipulatiag  objects _ 

Prefers  open  ended  questions 

ersonal  feelings 

■I  41  >4  4  4 


Free. 


ogic,  sees  correspondences,  resemblances 
Draws  o"  unbounded  patterns,  clustered  arouRd  images 


R 


igni  and  Left  Brain  Aciiviiiss 


Poor  reporting  structure 
The  Success  of  Any  Project  is  Linked  to  the  People 
Over-commitment 
Decision  Making  Escalation 
Political  Pressures 

Technology-focused  developments  -ignoring  the  Human 
The  lure  of  the  leading  edge 
Complexity  underestimated 
Inadequate  consultations  with  major  stake  holders^" 
Design  by  committee 

t  problem 


j  'zznnizctrn;rryrrrrcir&i% 
Staff  turn 


mover 


mcy 


I  ncif  lean 


_ _ .iiate  testing  and  training 

Communication  problems  and  work  group  problems 


sfclruf  Ri^ht 
Lack  of  Right  Brain 
Lack  of  Right  Brain 
Lack  of  Right  Brain 
Left  Brain  Knee  Jerk 
Lack  of  Right-Brain 
Lack  of  Right  Brain^, 
Lack  of  Right  Bcaim 
Lack  of  Left  Brain 
Lack  of  Right  Brain 
Lack  of  Right  Brain 
Lack  of  Right  Braingi 


i!S«ibr^rLrSrarrr 


Lack  of  Left  Brain- 
Lack  of  Right  Brlin 
Rck  of  Right  Brain3 
Lack  of  Right  Brain ^ 


rs  in 


iect  failures  are 


pr-gely-cFlaek  ofconnecting  issues  between 
right  and  left  brain: 


—User  involvement  /  Requirements  manaaemeat. 
-  Management  suffrpofj  rCo^jrfgplcafion 
Change  /  Adaptation 


(Marchewka,  2003) 


Project 

Size 

Avg 

Budget 

Avg 

Actual 

Cost 

Avg 

Over 

Budget 

Small 

$203K 

$435K 

214% 

Med 

$740K 

$1.36M 

1 82% 

Large 

$1.3M 

mm 

Avg 

Over 

Sched 

Original  Completed 
Scope  On-time 
Realized  On-budget 

Completed 
Over-budget 
Late,  not  all 
Functions 

239% 

74% 

28%** 

51% 

202% 

65% 

16% 

47% 

mm* 

9% 

62% 

Cancelled 

Before 

Completion 

22% 

37% 

30%^ 


i-and  left  brain 

-Und erstamf  how  we  truly  solve  problems, 
which  is  a  combination  of  both  sides 

Define  those  areas  of  Tight  braiq  gSTfVity  that 
wil  enhance  our  abilityjo  develop  systems  _ 

ly 


Ul 


H  urn 


m  Brain 


AncienijViodiaeval  and  Modern  Period 


_c4i  i y  i 

AristcrH 


^Piato ,  SoCratesr 


-  The  Development  of  Logic  and  Vertical  Thinking 

The  Bark  Ages  and  She  Renaissance!  CopeWcJQls 
to  Galileo 

-  Suppression  of  analytical  thinking 

Thfl  lnrl,jstrial  Revolution 

d  exDansio 


J 


93 


teral  Thin  kin 


Follows  leasPlkely 
paths 

Frequently  incorrect 
Non  sequential^-.* 


j 


thinking 

Bono,  1970) 


Vertical  Thinkin 

ftjlows  mosj  Ijkely 
path 

Always  correct 
Sequential 


StfTsiJ  vlwsl 


a 


J 


The  4  Golden  Ru 


specranyine'-more  experienced  /Thtelligeni  yoifare! 


Ask  Searching  questions 


To  gain  better  insight 


adopt  a  different  point 


l:. _  f< 


;  “What  If?” 


De  Bono’ 


6  Thinking  H 


es  we  miss  t 


to  receive  the 

-  Joseplfflenry 


rfraHrrrrrrarai.i  or 

mine 


iave  Blin 


(Wiseman,  2004) 
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Play 

Be  Present— 
'riak^heir  day 


frfc4 toefe 


PM 


1  P^«mnrL p  iilc8 
4  :ci  l>i  llmihl 
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\JF  [mp-rDfc*  K^t-uhi 


s  are 


gm 


posmg  a  different  way  (anno  re 
complete  way)  of  thinkingabote 

your  problem... 


-  Albert  Einstein 


Imagination  is  to  create 
not  exist  previously,  but  whjQh  cariUe 

ouah  Desttin  and  jj-ngineelarig 


exists 


memory1  or 


which 


CREATIVITY 


BEO-Gorpo rari o n  methods  of  success 
Deep  Survival 

SB  Surviving  is  a  Systems  solutiQjj^ 

Arts  and  Music  Education 


IT] 8  Arc  of  JnncK/acfon 


(Kelley,  2001) 


3Sf 


/TiriTTr^il 


nts  of  the  problem  (left  brain) 

Observe  real  people  in  real-life  (right  brain) 
Visualize  new  concepts  andl  the. customers 

I 

who  will  use  them  (right  brain) 

Evaluate  and«r«efine  the  prototypes  in  rapid 


-*  "  iiiZiir.  OQfl 

r-r», 


ew  corvee p 

commercialization  (both  sides) 


Stay  Calm 

Use  Humor,  Use  £earlo  focus  tftermiTTCi 


Deep  Survival 

(Gonzales.  2005) _ 


3.  Think/Analvze/Plan 


ake  Action 


Take  bold  afid  cautious. actions 


t 


:Take*joy-in-completing  "asks 

Count  youf  blessings 
Be  grateful,  help  others  more 


Play 


n 


o.  DSS  rns  09£)U[.v 


Treat  it  like  a  vision  quest 


■evefop^a-deep  conviction  of success 

.Surrender 

Leflgo  your  fear,  pujaway  pie  pain 

j  Do  whatever  is  necessaiy"^^"^ 

° ujelerjjjjneg).  tn^  will  andlne  skill 


SSk  yoorspiritr 

morelhing  yoiPcan  do 


one 


Take  Actiorr 

Celebrate  your 
successes 

Believe  you  will 
succeed 


j 


J 


iiEMBaaaaifl 


Never  give  up 


ght  Brain  Activities 

Perceive 

Stay  Calm 

Counl  youPblessihgs 
Play 

See  tne  Beauty 
Surrender 


Firs'Fgrotip  given  no  music  lessons  116% 
increase  in  spatial  reasoning  scores 


(SSecond  group  given  music  lesson<Sr9^6%, 
increase  in  spatial  reasoning  sdUTEk  ' 


acceptance 


i 


IMku 


who 


Kindergarten 


creation  qivetrl 


errormed  in 


E  After  one  year  were  22%  better  at:  mathematical 
competency  than  their  peers 

1993-95  study  on  three  groups  of 


*»znu  group  ^compiler  lessoEs*TQ^\3opts 
H3rd  group  -^jaajjsic  lessonsB  IQ  A  3.62  pts 


More  results 


J  J  J 


J 


ormance  experience 

-  53  points  higher  on  SAT  verbal  portion 

-  39  points  higher  on  SAT  math  portion 


-  Than  students  without  musical  performanc^xp^ 

■  Students  with  coursework  iatmysie-0'ppf«eciatioS 

-  61  points  higher  on  SAT  verbal  portion 


^e^Tlylnete  te  someming  of— 
significant  valleln  Whole-Brain 

Reasoning... 


jnclerstancling  a  System 


_i 


j 


j 


eavor  is  first  an 

l^rnostFcFmattefof  accurate1  seeing 

Beaming  to  really  see  what  is  in  Wont  of- us 
aljows  us  to  better  understand  tfi^worid^ 
around  us  and  the  essence,  of  our  problem 

eparture  from 


istic  an 


rrcBiiiiia  jn  a  cras.guric.i  ar.ciai 


tifltf  reasoning 


See-exaetiy  what  theprobfem  is 
Imagine  the  system  and  the  problem 


Imagine  the  solution  to  the  problem 
With  your  eyes  you  seewnat  isn’t  working,  you. 


have 


Whole  Brain  Reasoning 


j 


N^^afroT^l^t^et^ation^^ri^lvs  i  sTro 
the  left  side  of  the  brain 

Free  association  image  building  from  the 
right  side  of  the  brain 

This  is  a  departure  from  traditional  linear 


tauential-modi 


S 


TTOnBiwwirawra 


A  brand-new  edition  of  the  classic — 
expanded  and  updated 


rive  oasic  skills  or  drawing 


i  ne’peYb^ption  of  spaces 
The  perception  of  relationship 
The  perception  plights  incf^; 


on  of  the 


whole 


ion  of  context 


The  perception  of  relationship 

The  perception  oTllqht’MtlcIa 
ard  to  design  inte 


refCj 


^/vith 


create: 

-  Comprehensive 

-  Cohesive 

-  Creative 

■^Rrc^b  iHty4e-<so  pnbi  ne  tb 


mmsHE  if^r^irirLtgi^irLg^si^ 


verage  performed 
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lycVi 

If  * ■ 

■ 
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wg 

I- 

ij ,  jL 

I 

pin 

$fe: 

See  wl ryfyou  are  not  seeing,  cIBe  largely  to 

-  Incorrect  mental  models 

-  Lack  of  focus 

-  Lack  of  understanding 
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Edward  H.  Adelson 


Creativity  and  innovation 


te-ablfitylo  look  atine  same  thing, 
and'  to  see  something  completely. 

differently 


Dr.Haledjian  was  in  Inspector  Winters'  office  waiting  for  the  next  case  to 
solve  when  Inspector  Winters  came  in  with  Jacques  Strap.  "He's 
charged  with  murdering  Frank  Buzz,"  snapped  the  Inspector. 

"It's  a  mistake,"  cried  Jacques. 

"So  then  what  were  you  doing  in  an  alley  with  a  gun  and  a  dead  man?" 
asked  the  Inspector. 

"It  wasn't  my  gun  ,  and  I'll  tell  you  the  truth.  I  was  walking  past  the  theater 
when  I  saw  two  men  run  past  me.  The  second  man  was  carrying  a  gun. 
So  I  followed  them.  They  turned  into  an  alley  and  the  second  man  fired 
six  shots  at  the  other.  The  first  man  dropped  dead  to  the  ground.  The 
murderer  was  about  to  walk  away  when  he  caught  a  glimpse  of  me. 
Knowing  there  was  no  escape,  he  threw  his  Colt  .45  at  me  and  ran  to 
the  fire  exit  door  of  the  theater.  There  he  pushed  open  the  door  and 
went  inside.  I  was  still  holding  the  gun  when  a  cop  came  running  up  the 
alley,"  finished  Jacques. 

"I  tell  you  I'm  innocent." 

"Give  me  a  break",  said  Dr.Haledjian.  "This  story  is  impossible." 
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1 8”  deck  or  22”  deck? 

Mulching  blade  required? 

Do  you  want  variable  rale,  pop  iris  ioflff 
Side  or  rear  exhaust? 
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Triis  is  Systems  Engineering 


i  ne 


-  r—* 


ion  of  context 


The  perception  of  relationship 

The  perception  oTllqlrrMclTia 
uirements,  verifica 


rgc 


ge 


TSulTeqaires  us  to’ see  things  dUferenfly 

lAncfrequires  us  to  combine  elements  and’ 
perceive  the  system  in  ways,  not  nflffnally 
familiar  to  us 


m 


Dw  from  "the 


fP 


ucdami 


5m  a.whol 


75in  perspective 


S  u  mm  a  ry 


ng  and  Projects 

Ssii rts^T  Co n cr^WrAbs  tract  Thinking 
Seeing  Systems 
Whole-Brain  Thinking 


r T, 


HheJDevelop 


^ho  works'wiflrhis  hands  is  a 


H*rwho  works  with  his  hands  and  head  is  a 
craftsman 


He  who  works  with  his  hands, 
is  an  artistSg^^^^^^H 

St.  Fran 


heart 


De  Bono,  E  (1970)  Lateral  Thinking:  Creativity  Step  by  Step,  Harper  &  Row 
De  Bono,  E  (1999)  Six  Thinking  Hats,  Back  Bay  Books 

Defense  Science  Board  (1994)  Report  of  the  Defense  Science  Board  Task  Force  on  Acquiring 
Defense  Software  Commercially,  The  Undersecretary  of  Defense,  Acquisition  and  Technology 

Gonzales,  L  (2005)  Deep  Survival,  W.W.  Norton  and  Company,  London,  295  pp. 

Hoff,  B  (2002)  The  House  on  the  Point,  St.  Martins  Minotaur,  New  York,  New  York 
Jones,  C  (1996)  Patterns  of  Software  Systems  Failure  and  Success,  International  Computer  Press 
Kelley,  J.  (200f)  The  Art  of  Innovation.  Doubleday,  Random  House,  New  York^NY 
Lundin,  S.  et  al.  (2000)  Fish!  A  Remarkable  Way  to  Boost  Morale  and  l^roveJResultsnlyperioh 
Marchewka,  J.  (2003)  Information  Technology  Project  Management,  Wiley 
Murray,  P  (2003)  Systems  Thinking,  NDIA  Systems  Engineering  Conference,  San  Diego,  CA 
O’Connor,  J,  McDermott,  I  (1997)  The  Art  of  Systems  ThinkinJ’Thdrsons,  Hammersmith,  London 


LMI 

GOVERNMENT  CONSULTING 

THE  OPPORTUNITY  TO  MAKE  A  DIFFERENCE  HAS  NEVER  BEEN  GREATER 


BETWEEN  RELIABILITY  INVESTMENTS 
AND  LIFE-CYCLE  SUPPORT  COSTS 
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October  22-25,  2007 
San  Diego,  CA 
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Project  Phases 


•  Phase  I: 

-  Sponsored  by  DOT&E 

-  Using  empirical  data,  investigated  the  relationships  between 
reliability  investment  and  life  cycle  support  costs. 

-  Analyzed  the  root  causes  of  not  meeting  R&M  requirements. 

•  Phase  II: 

-  Co-sponsored  by  DOT&E  and  AT&L 

-  Building  on  results  from  phase  I,  developing  a  mathematical 
model  that  can  be  used  to  predict  the  investment  in  reliability 
required  to  achieve  a  given  amount  of  reliability 
improvement/growth. 


Phase  I 

Study  Approach 

•  Two  overarching  constructs 

•  Achieved  reliability  =  /  (goal  setting,  technology,  reliability  effort) 

•  Support  cost  =  /  (usage,  product  design,  support  process  design) 

•  Gap  analysis  between  the  earliest  and  the  most 
recent  data  available 

•  Statistical  analysis  of  reliability  investment  and 
reliability  improvement 
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Limitations 


Phase 


•  Data  availability  and  quality.  Problems  with  incomplete, 
corrupted,  and  missing  data  were  pervasive. 

•  Small  sample  size-empirical  data  limited  to  six  case 
studies. 

•  LCCs  were  estimated  by  CASA  model-not  actuals. 

•  Assumed  reliability  investments  were  the  cause  for 
observed  reliability  improvements. 

•  When  LRU-level  costs  were  not  available,  subsystem  level 
costs  were  allocated  to  the  LRU  level. 


CASA  =  Cost  Analysis  Strategy  Assessment  Model 
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Phase 


Bottom  Line  Up  Front 

•  Reliability  goals  did  not  appear  to  be  driving  either 
management  or  engineering  effort. 

•  Availability  and  use  of  mature  technology  was  not  an 
issue. 

•  Programs  significantly  improved  system  reliability. 

-  Improvement  ranged  from  50%  to  674.5% 

-  Improvements  were  partially  the  result  of  enhancements  to 
resolve  performance  limitations. 

-  In  four  of  five  cases,  programs  made  a  deliberate  effort  to 
improve  reliability  in  its  own  right — 2  of  4  cases  after  OT  or  IOC. 

•  Programs  significantly  reduced  life-cycle  support  costs. 

-  Percent  reductions  ranged  from  23%  to  86%. 

-  However,  under  investment  in  reliability  may  be  large  (discussed 
later). 
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Case  Studies 


Phase 


•  Predator 

•  Global  Hawk 

•  CH-47F  (characterized  improvement  but  did 
not  model  LCC) 

•  MH-60S 

•  FBCB2 

•  Complex  Ground  Vehicle  Electronics  System 
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rmr 

I  till 


Air  Force 
MQ-1  Predator 


Purpose: 

-  Develop  roug 

the  20-y 

of  achieved  reliability 

4 


te  return  on  reliability  investment 


Predator  Analysis 


Phase 


Reliability  History 


Reliability  Investment  and  Support  Cost  Reduction 


Case 

Study 

MTBx  Hours 

CASA  20-Year  Support  Cost 
(costs  in  FY03  M  $,  discounted  7% 

Economics  (FY03  M  $) 

Was 

Is 

Percent 

Change 

Was 

Is 

Percent 

Change 

Reliability 

Investment 

ROI 

Predator 

40 

77 

92.5% 

$1,463,940 

$576,663 

60.6% 

$39.1 

22.7:1 

I 
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rmy  FBCB  2 


Purpose: 


Develop  rough  order  of  magnitude 
(ROM)  estimate  of  the  20-year  life- 
cycle  cost  (LCC)  benefit  of  achieved 
reliability  improvement 


GOVEKPfMEfiT  COP'S 


Estimate  return  on  reliability 
investment 


LM 


MTBEFF  (Hours) 


Phase 


Global  Hawk,  MH-60,  and  FBCB2 


Global  Hawk  Reliability  History  ® 


Specification  Requirement: 
MTBCF>100  Hrs 


FY01 

FY02 

FY03 

FY04 

FY05 

FY06 

— MTBCF 

67.7 

95.7 

91.0 

114.2 

120.0 

117.1 

FBCB2  Reliability  History 


100 

50 

0 

ORD  (T)  Reliability  Goal 
(2002):  500  hrs  MTBEFF 

FY01 

FY02 

FY03 

FY04 

FBCB2  MTBEFF 

47 

121 

333 

364 

Reliability  Investment  and  Support  Cost  Reduction 


Case  Study 

MTBx  Hours 

CASA  20-Year  Support  Cost 
(costs  in  FY03  M  $.  discounted  7% 

Economics  (FY03M$) 

Was 

Is 

Percent 

Change 

Was 

Is 

Percent 

Change 

Reliability 

Investment 

R0I 

Global  Hawk 

67.7 

120 

77.3% 

$2,547,380 

$1,958,759 

23.1% 

$121.9 

5:1 

HMO 

2.4 

3.6 

*50% 

$384.(02 

$(4.(72 

83.2% 

$(.( 

40:1 

FBCB2 

47 

364 

674.5% 

$13.0(0.3(1 

$1,880.7(3 

85.6% 

$874 

128:1 

Improvement  over  HH-60  for  like  components 
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Ln  (Improvement  in  MTBx) 


Reliability  Improvement  vs.  Investment  Phase  I 

(Excluding  Complex  Ground  Vehicle  Electronics  System) 


Phase 


Translating  Reliability  Investment  Into  Support  Cost 
Reduction  (Complex  Ground  Vehicle  Electronic  System  Example) 


Potentially  technology  and 
application  neutral 


Most  likely  technology  and 
application  dependent 


20  Year  Support  Cost  Reduction  vs.  Reliability  Investment 
(Fraction  of  APUC) 


Complex  Ground  Vehicle  Electronics  System 


%  Reduction  in  Support  Cost 


Translating  Investment  as  Fraction  of 
APUC  Into  Support  Cost  Reduction 


Phase 
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Conclusions 


Phase 


•  Achieving  Reliability 

-  Goals 

•  All  five  case  studies  for  fielded  systems  had  established  goals 

•  ...but  did  not  manage  to  requirements 

-  Mature  technology:  available  and  incorporated 

-  Effort 

•  With  possible  exception  of  FBCB2,  reliability  effort  was  not  evident  during 
design 

•  Two  cases  with  reliability  effort  did  it  after  OT/IOC 

•  Reliability  and  ROI 

-  Clear  evidence  that  reliability  investment  reduces  support  cost 

•  ROIs  of  5:1  to  128:1  on  reliability  improvement  effort 

•  30%  to  80%  (or  more)  reduction  in  support  cost  achievable 

-  Reacting  to  unreliability  after  OT/IOC  rather  than  preventing  it  up-front 
is  probably  leaving  significant  support  cost  reduction  unrealized 
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Project  Phases 


•  Phase  I: 

-  Using  empirical  data,  investigate  the  relationships  between 
reliability  investment  and  life  cycle  support  costs. 

-  Analyze  the  root  causes  of  not  meeting  R&M  requirements. 

•  Phase  II: 

-  Building  on  results  from  phase  I,  develop  a  mathematical 
model  that  can  be  used  to  predict  the  investment  in  reliability 
required  to  achieve  a  given  amount  of  reliability 
improvement/growth. 
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Model  Comprises  Four  Sub¬ 
models 


Phase 


1.  A  basic  model  that  computes  reliability  development  effort  and 
cost  as  a  function  of  program  size  and  desired  reliability 
improvement 

2.  An  intermediate  model  that  computes  development  effort  as 
function  of  program  size,  desired  reliability  improvement,  and  a 
set  of  relevant  cost  drivers 

3.  A  detailed  model  that  incorporates  the  characteristics  of  the 
intermediate  version  with  an  assessment  of  the  cost  driver's 
impact  on  each  step  (analysis,  design,  etc.)  of  the  reliability 
engineering  process 

4.  A  companion  production/support  cost  model  to  estimate 

-  delta  investment  in  production  (e.g.,  for  retrofit  and  spare 
parts)  and 

-  change  in  operations  and  support  cost. 
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W1 

Study  Plan 


Phase 


•  Phase  lla  (DOT&E-funded) 

-  Basic  model 

•  Extension  of  work  done  under  cost  of  unreliability  task 

•  Robust  the  underlying  set  of  data  from  8  programs  and  projects  to  about  16 

•  Provide  cost  estimating  relationship  usable  by  program  managers,  including 

-  Applicability 

-  Statistical  properties 

-  Limitations 

-  Target  completion  February  2008 

•  Phase  Mb  (AT&L-funded) 

-  Intermediate  model 

-  Companion  production/support  cost  model 

-  Target  completion  June  2008 

•  Potential  follow-on 

-  Detailed  model...  if  warranted  by  results  of  phase  II 
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Phase  lla  Approach:  Focus  on  Phase  II 

Investment  to  Reliability  Relationship 


•  Extend  cost  of  unreliability  study, 
analyze  in-service  programs 

-  Reliability  improvement 

-  Cost  to  achieve  the  improvement 

•  Key  elements 

-  Emphasize  homogeneity  of  data 

•  Same  cost  content  across  programs 

•  Consistent  reliability  counting  rules 
within  a  program 

-  Look  for  disconfirming  evidence: 
programs  that  do  not  appear  to 
follow  the  log-log  relationship 
between  normalized  investment 
and  improvement,  reasons  why 

Cost  Content 

•  Funds  identified  for  reliability  improvement 

•  Funds  budgeted  for  test  related  to  reliability 
improvement 

•  Related  systems  engineering  funds 


Relationship 

Source  of  data/Comments 

Investment  to 

reliability 

improvement 

Across  air,  sea,  ground 
domains;  confirm  domain 
independence 

Reliability 

improvement  to  cost 
of  ownership 
reduction 

Single  domain  (air?); 
characterize  within  domain 

behavior 

Gather  data  in  phase  1,  limited 
analysis 

Summary 


•  Phase  I 

-  Using  empirical  data,  investigated  the  relationships  between 
reliability  investment  and  life  cycle  support  costs. 

-  Analyzed  the  root  causes  of  not  meeting  R&M  requirements 

•  Phase  II 

-  Extending  work  from  phase  I 

-  Developing  a  cost  estimating  relationship  usable  by  program 
managers,  including 

•  Applicability 

•  Statistical  properties 

•  Limitations 
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Additional  Slides 
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Reliability  Overview  by  Case  Study 


Phase 


Reliability  Goal 
Dimensions  0RD  (T) 

Fail  Rate 

Technology  Readiness  Management  Effort  Reduction 

Cases 

Predator 
(Air  Vehicle) 

40  hrs  MTBF 

•  Advanced  Concept  Technology  Demonstration  (ACTD) 
system  acquisition. 

•  After  initial  fielding,  ACTD  platform  upgraded  with  better 
performing  and  more  reliable  Engine,  Communications,  Flight 
controls,  and  Sensor  Payloads. 

•As  evidenced  in  budget  justifications,  concerted  emphasis 
was  placed  on  improving  overall  system  reliability  as  part  of 
improving  performance. 

48.1% 

Global  Hawk 
(Air  Vehicle) 

Effective  Time  on 
Station  (ETOS)  >  85%; 
MC>85%;  Translate  to 
MTBCF>100  Hrs 
(Spec) 

•  Advanced  Concept  Technology  Demonstration  (ACTD) 
system  acquisition. 

•  After  initial  fielding,  ACTD  platform  upgraded  in  Block  10 
spiral  with  better  performing  and  more  reliable  Electrical 

Power,  Communication  (Data  Link)  Improvements,  Flight 
controls,  and  Sensor  Management  Unit  (SMU)  configuration. 

•PM/OEM  data  and  budget  justifications  indicate  numerous 
trade  studies  and  other  engineering  initiatives  to  improve 
performance  and  reliability. 

43.6% 

Ch-47F 

The  original  goal  44 
hrs  MTBMA  was 
changed  in  2006  to  30 
hrs  MTBMA 

•  Remanufacture  of  CH-47D  to  Ch-47F  airframes  and  new 
zero  time  CH-47F  aircraft 

•  New  Platform  Airframe  to  reduce  vibration,  Common 
Avionics  Architecture  System  to  improve  avionics,  upgraded 

71 4B  Engine  with  enhanced  lift  capability  and  reliability, 

Engine  Air  Particle  Separator  to  improve  engine  reliability, 
and  reliable  Electrical  Power  Suoolv  and  Distribution  svstem 

•  PM  Improved  Cargo  Helicopter  (ICH)  contracted  with 

Avion,  INC  in  2002  to  baseline  and  manage  RAM  data 
collection  and  analyses. 

•  Failure  Modes  Effects  Analysis  (FEMA)  currently  being 
used  to  identify  reliability  bad  actors,  although  not  used  to 
drive  initial  design. 

35.8% 

MH-60S 

20.3  hrs  MTBOMF 
(~6hrs  MTBF) 

•  Remanufacture  of  HH-60  H  aircraft 

•  Investments  in  CPU,  PWR  SYS,  DRIVE  SHAFT  ASSY, 
NAV  COMPUTER,  STABILATOR  AMPLIFIER,  BEAM-AXLE 
AASY,  FLOOR  ASSY,  and  IR  TRANSMITTER  improvements 
have  improved  aircraft  performance  and  supportability. 

•  Overall  aircraft  reliability  has  shown  moderate  reliability 
gains  due  to  unreliability  of  older  subsystems. 

•  Did  not  find  specific  budget  justifcation  for  reliability 
improvement 

•  NAVAIR  PMA-299  is  working  with  the  Army  and  other 

Navy  programs  to  share  investment  expenses  to  improve 
reliability  of  bad  actors  such  as  retaining  bolts,  gaskets,  and 
bearings. 

33.3% 

FBCB2 

500  hrs  MTBEFF 

•  Beginning  with  tests  and  reliability  demos  on  competition 
systems,  investments  in  components  ruggedized  for 
operating  environments,  improved  power  supply,  redesign  of 
internal  components  to  dissipate  heat  and  removable  hard 
drives  and  dust  filters  were  made  to  improve  LRU  reliability 
and  overall  system  performance. 

•  Using  data  from  reliability  demonstrations,  field  tests  and 
operational  tests  and  analyses  to  drive  reliability 
improvement. 

•  As  evidenced  in  budget  justifications,  investments  are 
being  made  in  hardware  and  software  reliability 
improvements. 

87.1% 

Data  Availability  by  Platform/System 


Phase 


Platform 


MQ-1 

Predator 


RQ-4A  Global 
Hawk 


FBCB2 

Appliques 


MH-60S 


Usage  Data 
Density,  OPTEMPO 


Gaps  in  flying  hours  data  in  AFTOC. 
Useable  data  provided  by  MAJCOM. 


Gaps  in  data. 

Flying  hour  counting  rules  vary  by 
operational  unit.  (Beale  reports  pre  and 
post  flight  operations  in  FH  data;  Edwards 
does  not.) 


Product  Data 

Failure  rate,  MTBF,  MTTR,  MR 


Support  Process  Data 

CWT,  Cycle  Times,  NRTs 


Usable  data  available  from  OSD  Studies,  PM, 
and  other  sources 


Reliability  data  parameter  measured  in  the 
field  (MTBF)  not  consistent  with  the 
parameters  used  to  establish  ORD 
requirement  (ETOS>100  hrs) 


OT 


Usable  data  available  from  PMA  299 


Used  DT/OT  data  available  from  AEC  and 
IDA.  Data  differs  by  source  for  same  test 
events. 


Reliability  data  parameter  measured  in  the 
field  (MTBF)  not  consistent  with  the 
parameters  used  to  establish  ORD 
requirement  (MTBOMF) 


CH-47F 


OT  data  available  for  5  aircraft: 

2-EMD  Aircraft 
2-zero  time  CH-47F  Aircraft 
1 -Hybrid  Aircraft,  CH-47D  outfitted  with 
a  CAAS  cockpit 


Reliability  data  parameter  measured  in  1 
field  (MTBF)  and  ORD  requirement  (MTBMA) 
are  used  interchangeably.  Unclear  whether 
data  contain  all  failures  or  only  those  causing  | 
mission  abort. 
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Usable  data  available  from  standard  source 
Usable  data  available  from  PM  or  other  sources 

Data  problematic  incomplete  inconsistent  pedigree  suspect  or  unknown 


DEVELOPMENTAL  TEST  &  EVALUATION 


OSD  DT&E: 
Implementing  the 
Technology  Maturity  Vector 

Joe  Terlizzese 

Support  to 

Deputy  Director,  Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 

October  25,  2007 


DT&E  -  From  Concept  to  Combat 


Outline 


DEVELOPMENTAL  TEST  &  EVALUATION 

•  SSE  Mission 

•  Systems  Engineering  Revitalization 

•  Emerging  Systemic  Issues 

•  DT&E  Revitalization  Focus 

•  DT&E  Technology  Maturity  Initiative 

•  Pending  Guidance  Changes 


DT&E  -  From  Concept  to  Combat 


Systems  and  Software  Engineering 

Mission  Statement 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Shape  acquisition  solutions  and  promote  early  technical  planning 

•  Promote  the  application  of  sound  systems  and  software  engineering, 
developmental  test  and  evaluation,  and  related  technical  disciplines  across  the 
Department's  acquisition  community  and  programs 

•  Raise  awareness  of  the  importance  of  effective  systems  engineering  and  drive 
the  state-of-the-practice  into  program  planning  and  execution 

•  Establish  policy,  guidance,  best  practices,  education,  and  training  in 
collaboration  with  academia,  industry,  and  government  communities 

•  Provide  technical  insight  to  program  managers  and  leadership  to  support 
decision  making 


Evolving  System  Engineering  Challenges 


DT&E  -  From  Concept  to  Combat 


Systems  Engineering  Revitalization  Cycle 


DEVELOPMENTAL  TEST  &  EVALUATION 


DT&E  -  From  Concept  to  Combat 


Top  10  Emerging  Systemic  Issues 


DEVELOPMENTAL  TEST  &  EVALUATION 


1.  Management 

2.  Requirements 

3.  Systems  Engineering 

4.  Staffing 

5.  Reliability 

6.  Acquisition  Strategy 

7.  Schedule 


•  IPT  roles,  responsibilities,  authority,  poor  communication 

•  Inexperienced  staff,  lack  of  technical  expertise 

•  Creep/stability 

•  Tangible,  measurable,  testable 

•  Lack  of  a  rigorous  approach,  technical  expertise 

•  Process  compliance 

•  Inadequate  Government  program  office  staff 

•  Ambitious  growth  curves,  unrealistic  requirements 

•  Inadequate  “test  time”  for  statistical  calculations 

•  Competing  budget  priorities,  schedule-driven 

•  Contracting  issues,  poor  technical  assumptions 

•  Realism,  compression 


8.  Test  Planning 

9.  Software 


•  Breadth,  depth,  resources 

•  Architecture,  design/development  discipline 

•  Staffing/skill  levels,  organizational  competency  (process) 


10.  Maintainability/Logistics 


•  Sustainment  costs  not  fully  considered  (short-sighted) 

•  Supportability  considerations  traded 


Major  contributors  to  poor  program  performance 


DT&E  -  From  Concept  to  Combat 


DT&E  Revitalization  Focus 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Support  Faster  Fielding  of  Improved  Capabilities 

•  Reduce  Risk  of  Immature  Technology  in 
Systems  Development 

•  Revitalize  T&E  Workforce  Education 

•  Promote  Joint  T&E  in  Live-Virtual-Constructive 
Environments 

•  Provide  Effective  Acquisition  Policy  and  Practices  for  DT&E 


DT&E  Revitalization  Vectors 


DT&E  -  From  Concept  to  Combat 


Technology  vs.  Technical  Maturity 


DEVELOPMENTAL  TEST  &  EVALUATION 


DT&E  -  From  Concept  to  Combat 


Reduce  Risk  of  Immature  Technology  in 
Systems  Development 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Immature  technology  is  a  primary  source  of  cost  and  schedule 
risk 

-  GAO  -  DAPA 

-  QDR  --  SSE  Program  Support  Reviews 

•  “Programs  that  started  development  with  immature  technologies 
experienced  an  average  acquisition  unit  cost  increase  of  nearly 
21  percent”  (GAO-05-301,  March  2005) 


•  Milestone  B  -  USD(AT&L)  certification  that  " the  technology  in 
the  program  has  been  demonstrated  in  a  relevant  environment ”  - 
Technology  Readiness  Level  (TRL)  6  (FY06,  PL  109-163,  Section 
801) 


DT&E  -  From  Concept  to  Combat 


DT&E  Technology  Maturity  Initiative 


DEVELOPMENTAL  TEST  &  EVALUATION 


Purpose 

•  Add  Technology  Maturity  focus  into  the  Systems  Engineering  and 
DT&E  processes  to: 

-  Reduce  technical,  cost,  and  schedule  risk 

-  Increase  the  rigor  of  SE 

-  Plan  for  alternatives  in  the  event  of  TM  difficulty 

-  Verify  TRLs  during  DT&E 


Scope 

•  Leverage  existing  acquisition  review  structure 

•  Use  existing  DDR&E  Technology 

TRL  1 

Readiness  Assessment  (TRA) 
methodology 


MSA 


MS  B  CDR 


c/) 

c/5 

CO 

-C 

Q. 

E 

CD 


MS  C 


System  / 

Technical  Maturity 


DT&E  -  From  Concept  to  Combat 


time 


Pending  Guidance  Changes 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Defense  Acquisition  Guidebook 

-  Chapter  4  (SE) 

•  For  immature  critical  technology  elements  (CTE),  identify 
mature  alternative 

•  If  a  CTE  is  not  likely  to  reach  TRL  6  before  MS  B,  substitution 
of  the  mature  alternative  may  be  required 

-  Chapter  9  (T&E) 

•  Validate  technology  maturation  during  Technology 
Development  phase 

-  DT  supports  decisions  to  shift  to  alternative  technology 


DT&E  -  From  Concept  to  Combat 


Increased  TM  emphasis  in  OSD  Oversight 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Program  Support  Review  (PSR) 

-  ID  Critical  Technology  Elements? 

-  Current  TRLs  known? 

-  ID  Mature  alternative  components/sub-systems? 

-  TRL  monitoring,  Alternative  decision  date? 


•  Assessment  of  Operational  Test  Readiness  (AOTR) 

^sterns  Engineer;,^ 


-  TM  verification  results 

-  DT&E  performance  results 

-  IOT&E  predictive  analysis/M&S 


DT&E  -  From  Concept  to  Combat 


Sensor  Resource  Allocation  as  a  Driver 
in  System  Concept  Development 


Ravi  Moorthy 
Mark  Russell 
Jay  Davidson 
Lockheed  Martin  -  MS2 
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Topics 


•  Abstract 

•  Definitions 

•  Scenarios  and  Mission 

•  Multi  Mission  Scenario 

•  Sensor  Resource  Problem 

•  System  Considerations 

•  Engagement  Timeline 

•  Autonomous  Engagement 

•  Sensor  Resource  Management 

•  Event  Time  Preliminary  Analysis 

•  Event  Time  Derived  Analysis 

•  System  Concept  Development 


Abstract 


Operation  of  a  Primary  mission  concurrently  with  an 
added  mission  like  Anti  Air  Warfare  (AAW)  operations 
and  other  missions  requires  a  balancing  of  competing 
system  assets 

This  paper  provides  a  methodology  to  assess  the 
individual  mission  requirements  and  derive  the  capability 
assessment  of  multi  mission  scenarios 

This  paper  also  describes  the  drivers  for  such  multi 
mission  capability  of  the  Aegis  or  similar  naval  combat 
systems 

Overall  capability  to  protect  the  ship  and  other 
protected  assets  as  influenced  by  situation  dependent 
sharing  of  system  resources  and  other  system 
constraints  is  discussed 

A  brief  description  of  the  system  level  simulation 
model  is  also  provided 
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Definitions 


•  AAW  -  Anti  Air  Warfare 

•  Sensors  -  Scanning  and  Tracking  Radars 

•  Autonomous  -  Organic 

•  Multi  Mission  -  Combined  Missions  requiring 
Simultaneous  Tracking  and  engagements 

•  Radar  Resource  Usage  -  Radar  Time  required  to 
schedule  the  beams  for  search  and  track 
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Scenarios  and  Mission 


AAW Engagement  General  Timeline  Model 


Intercept 

fV 

Command 

Guidance 

Terminal 

Guidance 
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Multi  Mission  scenario 


•  A  ship  tasked  with  the  primary  mission  may 
need  almost  all  of  its  radar  resources 
dedicated  to  that  single  mission 

•  To  engage  in  an  AA  W  mission ,  it  would  be 
necessary  to  commit  another  ship  from  the 
fleet  operations 

•  Multiple  ships  will  be  required  to  perform 
multiple  missions 

-  Therefore  it  is  necessary  to  consider 
reallocating  the  radar  resources  to 
support  the  AA  W  mission 
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Multi  Mission  Scenario 


•  Capability  analysis  was  performed  of  the  extent  to 
which  various  levels  of  radar  availability  can  be 
effectively  utilized  to  defend  against  a  breadth  of 
AA  W  threat  scenarios 

•  The  corollary  to  this  is  the  effect  on  the  primary 
mission  performance  of  concurrent  AAW 
missions 
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System  Resource  Management  (iof2> 


•  Continuously  monitor  Multi  Mission  Sensor  Resource  usage 

•  Dynamically  re-allocate  Multi  Mission  resources  by 
predetermined  mission  priority 


Concept  1:  Fixed  Mission  Priority 


Concept  2:  Dynamic  Allocation 


Dynamic 

Allocation 
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System  Resource  Management  <2  of  2> 


•  Continuously  monitor  Multi  Mission  resource  usage 

•  Analyze  dynamic  schedule  to  avoid  resource  overload  in 
Multi  Mission  Scenarios 


Concept  1:  Fixed  Mission  Priority 


Concept  2:  Dynamic  Allocation 


Target  A 


Target  B 

Target  C 


4  Overload 


Target  A 


Target  B 

Target  C 


► 


Re  Schedule 

_ 1  1 _ 

-\ 

-► 

J - 1— ► 

No  Overload 
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Sensor  Resource  Management 


•  Continuously  monitor  sensor  resources  i.e. 
sensor  usage 

•  Dynamically  schedule  Multi  Mission  resource 
between  AAW and  Primary  mission  by  designated 
mission  priority 

•  Dynamically  reschedule  system  activity  to  avoid 
Multi  Mission  resource  overload 

•  No  fixed  capability  boundary  for  target  tracking 
and  engagement 

•  Preserve  Multi  Mission  system  resources  for 
committed  weapon  system  actions 
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Event  fi  men  ne 

(Concept  1:  Fixed  Mission  Priority) 


Event  2  Event  3 


Event  4  9  \  Event  5  Event  6 

— t- — I  i  I  — — ~r — 
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Event  Time  Line  (Concept  2:  Dynamic  Allocation) 


Simultaneous  Events 
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System  Concept  Development 


•  In  a  Multi  Mission  environment  engagement 
completion  is  based  on  sensor  resource 
scheduling  priority 

•  Event  times  are  delayed  to  allocate  needed 
resources  required  to  complete  the  event  timeline 

•  Missions  do  not  always  get  100%  allocation  of 
resources 
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Weapon  System  Support 
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So  What  is  the  Problem? 


Sustainment  environment  different 

-  Not  one  big  pass/fail  test 

-  Most  tests  associated  with  mods 


Our  organization  had  an  ad  hoc,  contractor  dependent, 
aircraft  unique  test  approach 

Instigated  a  step-by-step  Operating  Instruction 


-  Approach 

-  Management  k 

-  Expectations  \ 

-  Throughout  the  organization  ' 

Implemented  tangible  approach  that  is: 

-  Aimed  at  the  working  level 

-  Applicable  throughout  entire  organization 

-  Accounts  for  progress  through  metrics 

-  Always  starts  with  requirements 


Test  Process  Flowchart 


Step  1:  Build  an  Integrated  Test  Team  (ITT) 


•  Program  Manager  formally  establishes  ITT 
in  writing 

-  Standard  Letter 

•  ITT  consists  of,  at  a  minimum: 

-  Program  Manager 

-  Project  Engineer 

-  Center  Test  Authority 

-  Responsible  Test  Organization 

-  Representative  from  the  customer 

-  Representative  from  the  contractor 


Planning  Phase 
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Step  2: 


ew  Lessons  Learned 


•  Everyone  thinks  their  test  is  unique — but 
they  are  usually  wrong 

•  Review  established  lessons  learned  for: 

-  Quantifiable  criteria  (e.g.  noise) 

-  Testing  Techniques  (e.g  analysis,  M&S...) 

-  Test  Methods 


-  Previous  Problems 

-  Operational  Scenarios 


Planning  Phase 
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Step  3:  Define  Test  Requirements 

•  Review  established  Requirements 
Correlation  Matrix  (RCM) 

-  Ensures  test  requirements  has  direct  link  to 
source  requirements 

•  For  each  requirement  ask: 

-  Is  it  quantified? 

-  Is  it  verifiable/testable/measurable? 

-  What  verification  method? 

•  If  need  be,  send  requirements  back  to 
program  manager  for  clarification 

•  For  risky  verifications/testability,  send  risk 
to  Risk  Management  Team 


Planning  Phase 
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Define  Requirements 


•  Break  intial  requirements  down  into  a 
Requirements  Correlation  Matrix  (RCM): 

-  Spreadsheet  with  following  columns: 

*  Requirement 

*  Requirement  Source 

*  Derived  Requirements 

*  Quantification 

*  Operational  Conditions 

*  Initial  Risk  Assessment 

•  Give  RCM  to 

-  Test  Team  for  their  planning 

-  Risk  Mngt  Team  for  their  planning 
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Req 

Req 

Derived 

Req 

Op 

Risk 

Title 

Source 

Req 

Definition 

Quantification 

Cases 

(R/Y/G) 

Step  3:  Define  Test  Requirements 

•  Review  established  Requirements 
Correlation  Matrix  (RCM) 

-  Ensures  test  requirements  has  direct  link  to 
source  requirements 

•  For  each  requirement  ask: 

-  Is  it  quantified? 

-  Is  it  verifiable/testable/measurable? 

-  What  verification  method? 

•  If  need  be,  send  requirements  back  to 
program  manager  for  clarification 

•  For  risky  verifications/testability,  send  risk 
to  Risk  Management  Team 


Planning  Phase 
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Step  4:  Develop  Test  Metrics 


•  Three  minimum  metrics 

-  Test  Requirements  Metric 

-  Test  Risk  Management  Metric 

-  Deficiency  Report  Metric 

•  Required  only  during  the  Test  Execution  Phase 

•  Update  the  RCM 

•  Metrics  shown  to  management  at 
quarterly  Weapon  Systems  Review 

-  Shown  elsewhere  as  required  (PMRs,  PDRs, 
CDRs,  TIMS,  TRRs,  etc) 


Planning  Phase 


15 


Test  Requirements  Metric 


Test  Requirements  Metric 


□  Total  #  of  Requirements  □  Quantified  □  #  Verifiable  □  Resource  Assigned 
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Test  Risks  Management  Metric 


Test  Risks  Management  Metric 


Date  1  Date  2  Date  3  Date  4  TRR 


□  Low  □  Med  ■  High  ■  Closed  /  Mitigated 
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Deficiency  Metric  Report 


Deficiency  Report  Metric 
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Step  5:  Create  TES  or  TEMP 


•  Tailored  to  size  of  project 

•  Documents  strategy  for  conducting  test 

•  Documents  Roles  and  Responsibilities 

-  How  Redlines  handled 

-  How  DRs  handled 

-  Use  of  TIMs 

-  Scheduled  Test  Events  (TRB,  TRR,  etc..) 

-  Mishap  Accountability 

•  Rationale  for  test  verification  methods 
(inspection,  analysis,  demonstration, test) 


Planning  Phase 
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Step  6:  Integrate  Test  Plan  IMS  &  Funding 


•  Program  Manager  will: 

-  Ensure  the  test  program  schedule  in  the 
TES/TEMP  is  incorporated  into  IMS 

-  Work  with  contractor’s  processes/timelines — 
not  duplicative 

-  Ensure  appropriate  test  program  funds  are 
available  to  support  TES/TEMP 

-  Schedule  technical  interchange  meetings  as 
required 


End  of  Planning  Phase 
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Step  7:  Technical  Reviews 


*  Testing  Addressed  in  Periodic  Reviews 

-  System  Requirements  Review 

-  System  Design  Review 

-  Preliminary  Design  Review 

-  Critical  Design  Review 

-  Safety  Reviews 

*  ITT  meets  periodically  to  review  that  all 
requirements  are: 

-  Tested 

-  Quantified 

-  Verifiable/testable/measureable 

-  Resourced 

-  Risks  mitigated 


Design  Phase 
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Step  8:  Update  TES/TEMP 


•  Update  at,  or  immediately  after,  each 
review 

•  Update  RCM  as  required 

•  Update  all  metrics 


Step  9:  Test  Readiness  Review  (TRR) 


•  TRR  required  before  any  formal  test 

•  01  has  a  clear  checklist  for  TRR 

-  Approved  test  procedures 

-  Test  scheduled  defined 

-  Hardware  installation  complete 

-  Software  configuration  is  stable  (passed  FQT) 

-  Support  requirements  defined  and  scheduled 

-  Test  team  identified 

-  User  training  integrated 

-  Mishap  accountability  identified 
—  Etc. 


Execution  Phase 


Step  10:  Test  Execution 


•  Execute  the  Test 

•  Document  Deficiencies 

-  Important  to  have  a  formal  process 

-  Hold  deficiency  reviews 

-  Correct  deficiencies 

-  Retest  the  system 


Execution  Phase 


DR  Quad  Chart 


E-4B  1677  MB1  Deficiencies 

VHF/FM  Red  to  Black  Audio  (DRB-139) 


Deficiency  -  Category  1 

Technical  O 

Description 

•  During  transmissions  via  VHF/FM  through  the  Black 

Switch  w/  the  radio  in  secure  mode  the  signal  bleeds 
over  onto  the  unsecure  channel 

•  Not  E-4  unique  issue 

-  Proposed  solution  part  of  s/w  release  for  all  fielded 
radios 

Actions  to  date 

•  Identified  after  the  installation  of  the  new  VHF/FM  radio 

-  Issue  identified  to  radio  manufacturer  (Wulfsberg) 

-  Wulfsberg  identified  a  s/w  solution 

•  Minor  software  anomalies  discovered  in  prototype  testing 

Requirement 

•  Derived  security/certification  requirements 

Way  Ahead 

•  Wulfsberg  setting  up  representative  test  lab 

•  Scheduled  to  complete  lab  testing  by  9  Dec  05 

Exit  Criteria 

•  Transmit  via  VHF/FM  through  the  Black  Switch  w/the 

radio  in  secure  mode  without  the  signal  bleeding  over 
onto  the  unsecure  channel 

•  A/C  integration  testing  scheduled  by  16  Dec  05 

Technical  POC:  Jim  Barnabv 

Funding  O 

Schedule  O 

Funding:  Solution  covered  under  warranty 

Aggressive:  28  Nov  05 

Moderate:  6  Dec  05 

POC:  John  Smith  (E-4  SPO)  DSN:  336-2547 

Low  Risk:  6  Jan  06 

Updated  16  Nov  05  CU 

Step  11:  Test  Report  and  Lessons  Learned 


•  Tests  are  not  snowflakes 

•  Lessons  Learned  repository  contains: 

-  Possible  tests  to  consider 

-  Potential  test  plans 

•  Repository  is  not  program  specific,  but  for 
entire  organization 

•  Future  plans  are  to  make  the  lessons 
learned  repository  a  database  with 
keyword  searches 


Execution  Phase 
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What’s  Next 


Continue  implementation  throughout 
organization 

Continue  Measure/Track  results 
Populate  Lessons  Learned  database 
Refine  as  needed 
Document  successes 


-  We  are  having  some! 


Test  Management  can  be  implemented,  applied 

AND  make  a  difference 
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Summary 


•  727th  ACSG  developed  grass-roots  means  to 
implement  Test  Management  as  part  or  our 
Systems  Engineering  in  Sustainment 
Environment 

•  Clear-cut,  tangible  processes  steps  for  the 
working-level 

•  Metrics  to  measure  progress  for  management 


I  n  Place  and  I  n  Use  Now 
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Questions? 
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Basic  Systems  Engineering  Process 
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Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 
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Design 
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Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Agile  Governance  for  SOA-Based 
Military  Systems  of  Systems 


Robert  Beck,  Jessica  Byrnes, 
Sue  Metzger,  and  Elliot  Sloane 
Villanova  University 


Outline 
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Background 

Net  Centric  operations 

Service  Oriented  Architecture  (SOA) 

-  Modeling  and  Simulation 

Systems  of  Systems  (SoS) 

Co-evolutionary  development 

-  Agile  development  strategy 

Governance 

-  Agile  Governance  for  SOA  (AGSOA) 


10th  Annual  Systems  Engineering  Conference 
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fSg|VI  LLANOVA 

UNIVERSITY 

Background 

ARCES  Program  (Applied  Research  in 
Computing  Enterprise  Services) 

-  Research  contract:  AF  ESC  to  Villanova 

Addresses  inhibitors  and  threats  to  emerging 
SHARED  military  IT  applications 

-  Modeling  and  simulation 

-  Compression 

-  Communities  of  interest;  granularity  of  information 

-  Security 

-  Strategies  for  effective  co-evolutionary  development 


10th  Annual  Systems  Engineering  Conference 
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Presentation  Focus 

Not  modeling  and  simulation 

-  Very  interesting  and  successful  results 

-  Net-Centric  Validation  Conference 

•  September  27-28,  2007  at  Villanova 

But  why  some  Net-Centric  warfare 
developments  are  not  moving  as  rapidly  as 
expected 

And  how  to  deal  with  the  problem 


LLANOVA 

IVERSITY 


10th  Annual  Systems  Engineering  Conference 
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Net  Centric  Warfare 


Metadata! 

Registries 


Community 

Metadata 

Catalogs 


Shared  Data 
Storage 


Web -Services 
infrastructure 


"Just  finished  the 
analysts  report  3 rid 
want  lo  post  (1  tor 
people  to  see' 


utilizes  cataloging  system  to  post 
End-User  Producer  analysis'  report  and  provides 
metadata  about  it. 


want  lo  find  out 
howto  read  in 
collection 
targeting 
information" 


The  metadata  prodded  is  driven  by 
wrist  C  Ot  [s )  the  user  joins  Th  e 
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End-User  Consumer 


System 


Searches  community  m  etadata 
catalogs  to  Tina  products  (e  g 
Godgle  or  Yahoo 

Reads  metadata  Information  to 
understand  context  or  products 
round 

Pulls  needed  data  based  on  the 
metad^a 


End-User  Developer/Designer 

During  the  development  stage* 
the  m  eta  data  about  standard 
data  and  document  formats  is 
pasted  and  retneved  from 
metadata  registries  for  reuse 


Systems  nave  been 
programmed  to  conform  to  data 
schemas  defined  in  the 
Metadata  registry 


Ubiquitous  Network  and  Services  that  support  the  "enterprise 
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university 

Net  Centricity  — >  SOA 

•  Net  centricity  is  evolving  to  the  more  fluid 
service  oriented  architecture  (SOA  )  based 
systems 

However, 

•  Net  centricity  does  not  imply  SOA 

-  Can  be  server  to  server  based 

-  Can  be  client/server  based 


10th  Annual  Systems  Engineering  Conference 
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A  SOA  Primer 


VILLANOVA 

UNIVERSITY 


Naturally  systems  of  systems 
Complex  interdependencies 

Require  overarching  governance  structure 
-  Better  imagined  as  a  foundation 

Distributed  development 

Co-evolutionary  (multiple  overlapping 
spirals  with  short  time  frames) 


10th  Annual  Systems  Engineering  Conference 
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Understanding  Co-Evolutionary 

Development 


"  \ 

Co-Evolutionary 

Development 


\ _ / 

\ 

uses  concepts  from 


10th  Annual  Systems  Engineering  Conference 


8 
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A  SOA  Primer  (2) 

Example:  Create  a  delivery  tracking  and  support 
system  through  a  combination  of  services 

-  Traffic.com 

-  Google  Earth 

-  Mom’s  Mileage  Management  (MMMgood.biz) 

Notes 

-  Delivery  service  depends  on  real-time  view 

-  Shared  resource 

-  Large  number  of  simultaneous  consumers  /  providers 


10th  Annual  Systems  Engineering  Conference 
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Engineering  Challenge 

By  definition,  SOA  systems  are 

-  interdependent 

-  greater  than  the  sum  of  the  parts 

But  interdependence  creates 

-  novel  risks  to  stakeholders 

-  especially  to  service  providers 

•  difficult  to  measure  and  reward  performance, 
value,  or  blame 

•  difficult  to  troubleshoot 

•  difficult  to  resolve  problems 


SOA’s  Systems  of  Systems 


10th  Annual  Systems  Engineering  Conference 
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SOA-based  Military  Systems 

Publish  and  subscribe  paradigm 

Perpetual  interdependencies 

AF-ICE  (Integrated  Collaborative 
Environment) 


10th  Annual  Systems  Engineering  Conference 


11 


ISgfVILLANOVA 

university 

Systems  Integration  Issues 

•  Complex,  multi-owner  systems  of  systems 

•  Emergent  requirements 

•  Rapid  change 

•  Reused  components 

•  High  assurance  of  qualities  of  safety,  security, 
reliability,  availability,  maintainability, 
performance,  adaptability,  interoperability, 
usability,  scalability 


See  references:  Barry  Boehm,  October,  2007 


10th  Annual  Systems  Engineering  Conference 
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SOA  Governance 

Actions  taken  to 

•  establish 

•  interpret 

•  enforce 
Rules  for 

•  development,  which  is  distributed 

•  implementation,  which  requires  cooperation  of 
many  natural  competitors  (co-opetition) 

•  operation,  which  depends  on  services  managed 
by  many  parties 


10th  Annual  Systems  Engineering  Conference 


13 


VILLANOVA 

UNIVERSITY 


Done  for  E 
or  Major  (p 


I, 


within  Entt< 
Manages 


rpr 

an 


pri 

or 


isle  and/ 
ization 
Se  that 

■mi  of 


10th  Annual 'systems 'Errgir 

Investments 


Conference 


Governance  as 
foundation 
vs. 


Governance^epfg 
supervision 


Cyc 


Federate 

Strc 


Feder$te< 

Arrh 


flg|VlLLANOVA 

UNIVERSITY 

SOA  Governance  Challenges/Tasks 

•  Security  infrastructure 

•  Service  versioning 

•  Service  funding 

•  Co-evolutionary  development 

•  Performance  monitoring  and  load  balancing 

•  Business  process  re-engineering 

•  Business  continuity  planning 

•  Disaster  recovery 
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Agile  Governance  for  SOA 

Constant  incremental  change  management 
and  improvement  in  governance  actions 

30-day  sprints 

Constant  communication  with  stakeholders 
Stakeholders  are: 

-  Service  providers 

-  Large  system  integrators 

-  Small  system  integrators 

-  System  users 
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AGSOA  Influence 

Provides  rules  and  services  that  are 
constantly  being  adapted  to 

-  Changes  in  technology 

-  Changes  in  customer  needs 

-  Changes  in  external  environment 

-  Changes  in  the  installed  and  documented 
base  of  software 
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UNIVERSITY 

AGSOA 

Governance  as  a  Service 

-  to  multitude  of  projects 

-  vetting  of  standards 

-  qualifying  open  source  and  off-the-shelf 
systems 

Compare  with  Software  as  a  Service 
(SaaS) 
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UNIVERSITY 

Effective  Governance  Components 

•  Adaptation  to  changing  social,  cultural, 
technical  and  business  complexities 

-  Gmail  vs.  Hotmail 

•  Trust  management  and  resolution  for 
stakeholders 

-  Intellectual  property  sharing  and  valuation 

•  Conflict  arbitration  and  resolution 

-  Reaching  agreement  on  platform  releases  for 
widely  varied  requirements 
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Example:  AGSOA  for  Security 


Sets  rules 
Collect  metrics 

-  for  incentives, 
accountability  and  risk 
management 

Contracts  services 


AGSOA  for  Security 
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AGSOA  may  demand  expertise- 
and  resource-sharing  across  business  and 
technical  boundaries 


VI  LLANOVA 

UNIVERSITY 


Multiple  simultaneous 
AGSOA  teams  likely 

A  “super-team”  may  be 
dynamically  created  to 
solve  an  interim 
problem  with 
resources  drawn  from 
other  teams 

Resources  expected  to 
be  returned  when  task 
is  accomplished 


AGSOA  Team 


AGSOA  (taskB)  AGSOA 


Team 

(task  C) 
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Team 
(task  A) 

10th  Annual  Systems  Engineering  Conference 


#5§2fJ  V I L  L  AN  OVA 

university 

Why  Agile  Governance  for  SOA? 

Tasks/goals  of  Net-Centric  Warfare  are  agile 

-to  meet  rapid  enemy  re-configuration 

-  to  seize  emergent  targets  of  opportunity 

Traditional  SOA  Governance  (for  Retail  and 
Manufacturing)  presumes 

-  longer  time  frames  (months  vs.  days) 

-  more  stable  goals 

-  better  defined  rewards 
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Why  Agile  (2) 


VILLANOVA 

UNIVERSITY 


Agile  SOA  governance  assumes 
governance  tasks,  priorities,  and  resources 

-  will  be  tuned  at  least  as  frequently  as  Net- 
Centric  Warfare  goals  change 

-  will  accommodate  unexpected  technology 
problems 

•  capacity  overload 

•  patch  bugs 
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AGSOA  Examples 

Porting  Tri-Services  IM  and  Email  to 

iPhone 

-  New  contracting  (now  dealing  with  Apple) 

-  Continuing  V&V  (responding  to  frequent 
iPhone  revisions) 

Shifting  secure  “Munitions  on  Target” 

resources  to  shared  SOA  platforms 

-  Contracting  with  dozens  of  contractors  for 
interface  design  and  V&V  that  affects 
hundreds  of  mission  critical  platforms 
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AGSOA  Governance  Services 


Examples  include: 

•  Providing  contract  negotiation  across  many  disparate  applications 
that  need  to  share  a  common  resource 

-  Identity  management  systems,  for  example 

•  Integrating  large  numbers  of  disparate  simultaneous  projects  and 
project  milestones  that  cross  vendor  boundaries  but  must  work 
together  in  a  SOA  system 

•  Managing  shared  Software  as  a  Service  resources  to  ensure  that 
stakeholders  are  fairly  compensated  or  charged  in  an  accurate  or 
timely  fashion 

Different  projects  will  need  and  consume  differing  amounts 
of  AGSOA  services  at  differing  times  in  their  project 
lifecycle,  centralized  and  shared  for  efficiency,  but 
AGILE  to  meet  unique  needs. 
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Presentation  Outline 
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•  Overview  of  Service  Oriented  Architecture  (SOA) 

-  Common  Reference  Architecture 

•  Methods  of  migrating  from  System  of  Systems  PORs  to  SOA 

•  Program  acquisition  costs  for.  migration  of  POR’s  to  SOA 

•  Mitigation  of  acquisition  costs 

•  Program  Examples 

-  Global  Combat  Support  System  -  Air  Force  (GCSS-AF) 

-  An  Army  Program 
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Common  Reference  Architecture 
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Requirements 


Expertise I 
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Service  Management 


•Business  or  Mission  Enablement  Element  - 
Supports  the  definition,  modeling,  and  analysis  of 
services  and  service-oriented  architectures  that 
meet  the  business  /  mission  needs 

•This  element  of  the  architecture  will  be  updated  as 
the  SOA  is  in  use  to  optimize  the  performance  of 
the  services  for  accomplishment  of  a  desired 
mission 

•As  new  services  come  online  the  system  will 
continuously  be  redesigned  to  take  advantage  of 
the  latest  technologies  oeing  employed. 
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Service  Creation 


LOCKHEED  M A R T I N / 


Service  Creation  Element  -  Service  creation 
encompasses  the  engineering  tasks  required 
to  develop  a  new  service,  incorporate  an 
existing  service,  or  to  convert  a  legacy 
application  into  a  service 
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Services  Container 
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•  Services  Container  Element  -  Assembly  of  services  based 
on  business  /  mission  enablement  processes  and 
supported  by  an  infrastructure^framework.  Various 
combinations  of  services,  business  logic,  andjworkflow 
are  combined  to  provide  service  avaifebiljty  to  the  COIs 
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Infrastructure  Services 
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Infrastructure  Services 


Infrastructure  Services  -  The  basic;  underlying 
services  that  enable  the  accessibility, 
interactions,  communications,  and  runtime 
operations  of  a  SOA 


-  Security 

-  Mediation 


-  Service  Management 

-  Information  Delivery 

-  Discovery 

-  Data  Management 
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Supporting  Infrastructure 


LOCKHEED  M  A  R  T  I  N  / 


•Supporting  Infrastructure  -  Products  and 
technologies  used  in  the  storing,  transmitting  and 
utilizing  of  information  that  supports  the  runtime 
operations  of  a  SOA 


Service  Bus 

Registry 

Repository 

System  Management 

BPEL  Engine 

Collaboration 

Application  Services 

Web  Support 

Transport 

Storage 


Supporting  Infrastructure 
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•  Service  Management  -  Configuring, 
monitoring,  maintaining  and  managing  of 
services  and  the  service  infrastructure 

-  Business  Activity  &  Event  Monitoring 

-  Fault  Management 

-  Configuration  Management 

-  Service  Level  Agreement  Management 

-  Performance  Management 
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Security 


LOCKHEED  MARTIN 


Security 


Security  -  Mechanisms  which  protect  e 
within  the  SOA  environment,  including 
consumer,  provider,  services  and  information; 
and,  additionally  provide  interface  between 
security  services  from  different  enterprises 


-  Policy  Management 

-  Assured  Information  Sharing 

-  Availability 

-  Network  Defense 
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Service  Management 

Security 


ISO, 


Governance 
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Governance 


Governance  -  Collection  and  enforcement  of 
policies  which  the  enterprise,  services,  and  data 
owners  must  abide  by  to  guarantee  success  at  all 
tiers  of  the  SOA 

-  Design  and  development  practices 

-  Technology  selecfiongM 

-  Policy  managemeiOT^H^ 

-  Contract  management  ^ 

-  Configuration  management 

-  Release  managemdafl^H 

-  Runtime  management 

-  Problem  maiagemenUaH 

-  Change  management 


Fill 

.1  Applies 

Servic 

Service 

Creation  Services  „ 

r  — 

■ : :  ■-  - I  Deployment  Architecture 


_  ~  infrastructure  At 

Supporting  Infrastructure 
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Migration  Methods 


LOCKHEED  M  A  R  T  I  N  / 


•  Build  Infrastructure  -  Leave  POR  in  Place 

•  Build  some  infrastructure  for  new  services^wrappers  on 
non-SOA  components 

•  Build  services  for  certain  components  gjadd  infrastructure 
later 

•  Wait  until  infrastructure  is  in  place  a  build  services  that  work 
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§h»  .  Transformation  Use  Case:  Life-Cycle  Levels  of 
_  Integration  (LLI)  Continuum 


LOCKHEED  MARTIN 


Level  1 


Level 


z 


z 


Level  4 


Level  5 


Discoverable  Services 

•  Context  independence 

•  Constraints  on  how  Components  are  built 

•  Bus-based 


Interoperable 

•  Identity  Mgmt;  Data 
Exchange;  Method  Exchange 

•  Interface  Registration 
Constraints 


Bus-Based 
Implementation 
Interface  Payload  Control 
Containers,  Connectors 


Interconnected  (All  types) 

•  Local  Service  Registry  becomes  Service  Proxy 

•  Hub-based  implementation  (Runtime  Interface  Registry;  Dispatcher) 

•  Full  SOA  Design  and  Runtime  Documentation  Required 


Controlled  (Enterprise  Configuration  Management) 


Still  point-to-point 

No  hard-coded  interface  connections 
Common  Local  Service  Registry  as  a  config.  file 


Common  Interface  Design  time 
documentation 

Basic  Design  time  information 


Specified 

•  Common  Specification  Format 

•  Control  over  the  Service  Components  Built 

•  Basic  Design-time  Documentation 


No  control  over  runtime  implementation 
Point-to-point,  program-to-program 


No  integrated  interface  information 


Inregnwon  jo  si  eer  or  enterprise  crmrecrerisues 


These  character!  sties  are  addltivei  Level  £>  integration  oormdno  Level 4;  Level 4  eonmlne  Level  3;  „ 
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Transformation  Use  Case:  Legacy  System  Enterprise  Application  Integration 


Mow  does  one  imsgram  legacy  applications  and  dam? 

-  For  purposes  of  this  use  case  assume  the  following: 

Current  Enterprise  State  that  is  at  Level  2 
or  below  on  the  LU  Continuum 

Program  centric  Domain  expertise  &  huge 
legacy  portfolio  (S/W  development) 

Multiple  SOA  systems;  not  stovepipe  -> 
reuse  opportunity,  maturity  models, 
guidelines 

-  Apply  integration  strategies  to  system  components 

that  meet  the  criteria  — 


Requirements 
Expertise  Conops 


Business  /  |l| _ 

Mission  /  Measures 
Enablement 

Processes  


Communities  of  Interest  fCP,cl 


Services  Availability 


Pipeline  -g 


Service  -LU - 

Creation  Services 


Orchestrated 

Services 


Infrastructure  Services 


Q  Infrastructure  A] 

Deployment  Architecture  Supporting  Infrastructure 


SOA  End  State  (Legacy-Focused) 
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Inter-Enterprise 
Business 
Services 


Enterprise  SOA  Infrastructure 
/Richer  but  not  pervasive,  more  governance 
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Enterprise 

Infrastructure 


Enterprise 
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Services 
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Business 
Applications 
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Heritage 
Business 
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Ind  System 
SOA  Infrastructure 


Heritage 


Business 
Applications 


Ind  System  El 
Infrastructure 
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Partner  Enterprise  Y 


i  Ind  System  lx 
Business 

7=|  Services  j 


Multiple  Tiers  allow  incremental  SOA  advancement 
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Software  Cost  Drivers  for  legacy  System  Migration 

•  Operating  Systems  -  upgrades,  licensing  fees 

•  COTS  Products  -  licensing  fees,  maintenance  contracts 

•  DBMS  -  licensing  fees,  maintenance  contracts 

•  GOTS  Products  -  upgrade  schedule  -  POM  Cycle 

•  New  Development  -  POM  Cycle,  JCIDS 

•  SOA  Infrastructure  Development 
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Mitigation  of  Software  Costs 

•  Operating  Systems  -  Upgrades,  Licensing  Fees 

-  Consolidation  on  fewer  OS’s 

•  COTS  Products  -  Licensing  Fees,  Maintenance  Contracts 

-  Enterprise  licensing 

•  DBMS  -  Licensing  Fees,  Maintenance  Contracts 

-  Data  Access  Service  =>  Reduce  DBMS’s 

•  GOTS  Products  -  Upgrade  Schedule  -  POM  Cycle 

-  Wrappers  Standardize  Interfaces 

•  New  Development  -  New  Requirements,  JCIDS  Process 

-  New  WS  reduce  deployment  cycle  time  -  Portfflmay  address  reqmnts 

•  SOA  Infrastructure  Development 

-  Tiered  architecture  reduces  initial  investment 
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Program  Examples 
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•  The  following  examples  show  the  degree  to  which  the  principles 
discussed  thus  far  have  been  implemented  and  how  effective  these 
efforts  have  been  to  date 


-  Global  Combat  Support  System  -  Air  Force  (GCSS-AF)  -  SOA  M 
migration  project  waslnjtiated  in  2003 

-  An  Army  Program  -  Migrating  to  SOA  and  scheduled  to  deploy  toljaq 
in  2009. 
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GCSS-.AF 

https://www.gcss-af.com/cfs/outreach/index.cfm 
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GCSS-AF  SOA  Journey 
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Stage  Name 

Business  Silos 

Standardized 

Technology 

Optimized  Core 

Business  1 

Modularity  1 

IT  Capability 

Local  IT 

Applications 

Shared  Technical 

Platforms 

Enterprise -wide 

Hardwired  Processes 

and  Data 

Plug  and  Play 

Business  Process 

Modules 

Business 

Objectives 

ROI  of  Local 

Business  Objectives 

Reduced  IT  Costs 

Cost  and  Quality  of 
Business  Operations 

Speed  to  Market 
and  Strategic 

Agility 

Funding 

Individual 

Applications 

Shared  Infrastructure 

Services 

Enterprise 

Applications  and  Data 

Stores 

Reusable  Business 

Process 

Components 

Key 

Technology -Enabled 

Design  and  Update  of 

Core  Enterprise 

Management  of 

Management 

Change 

Standards;  Funding 

Process  Definition  and 

Reusable  Business 

Capability 

Management 

Shared  Services 

Measurement 

Processes 

Who  Defines 

Local  Business 

IT  and  Business  Unit 

Senior  Management 

IT,  Business  and 

Applications 

Leaders 

Leaders 

and  Process  Leaders 

Industry  Leaders 

Key  IT 

Governance 

Issues 

Measure  and 

Communicate  Value 

Global  Responsibilities 

Align  Project 

Priorities  with 

Architecture 

Objectives 

Define,  Source  and 
Fund  Business 

Modules 

|  Source:  Architecture  as  Strategy  by  Ross 

,  Weiff,  Robertson ,  MBS  Press,  June  2006 

https://www.qcss-af.com/noosphere/space/SOA+Jou 
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Web  Page  by  John  Wunder,  Lockheed  Martin  Corporation  Copyright  20o|i 
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*  •>  ROI  of  Local  Business  Objectives: 
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•  ROI  is  sub-optimized  to  within  program,  unit,  lifecycle  boundariesjlnterfaces 
optimized  to  specific  method  needs  reducing  by  bytes  size  of  message  while 
proliferating  protocols  leading  to  hundreds  of  different  interfaces  on  dozens  of 
protocols  burdening  source  systems  interface  code  that  is  over  70%  of  their 
function,  causing  exceedingly  high  operations  cost  and  setting  up  a  closely 
coupled  brittle  structure  that  is  exceeding  expensive  to  move  forward.  Each  phase 
of  the  lifecycle  optimizes  for  itself.  Shorter  delivery  schedules  or  cheaper  product 
costs  are  not  considered  in  relation  downstream  impacts 
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Reduced  IT  Costs: 


LOCKHEED  M  A  R  T  I  N  / 


•  Commoditization  of  IT  Infrastructure  Enterprise  licenses  for  software  and 
hardware.  Reduced  O&S  cost  through  common  infrastructure.  Reduced 
assembly  cost  through  eliminating  'botique  engineering' 


Copyright  2007  by  Lockheed  Martin  Corporation 


22 


W-  Cost  &  Quality  of  Business  Operations: 


LOCKHEED  M  A  R  T  I  N  / 


•  BEA  (DoD  Business  Enterprise  Architecture)  eLog21  and  other  like  minded 
functional  initiatives  focused  on  mission  processes,  people  and  information  and 
measured  success  not  in  lines  of  code,  deliveries  or  response  time  but  rather  in 
mission  effectivenes  and  asset  status,  particularly  aircraft. 
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Strategic  Agility: 


LOCKHEED  M  A  R  T  I  N  / 


•  Velocity,  Velocity,  Velocity  was  our  mantra.  It  allowed  us  to  make  and  correct 
mistakes  faster  and  with  the  built  in  feedback  we  had  we  consistently  moved 
forward.  Have  not  reached  the  Chief's  vision  of  changing  information  flows  in 
hours  and  days  but  it  is  on  the  horizon! 
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Individual  Applications: 


LOCKHEED  M  A  R  T  I  N  / 


•  We  were  going  to  modernize  all  applications  and  every  modernization  was  a 
separate  program.  'Silver  bullet'  approaches  program  by  programgSpecific 
examples  were: 

-  1.  GOLD  for  IMDS  and  SBSS 

-  2.  Teradatafor  FIRST 

-  3.  AF  Portal  for  OLVIMS 

-  4.  J2EE  for  CAS 

-  5.  Homegrown  Portal 
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Shared  Infrastructure  Services: 


LOCKHEED  M  A  R  T  I  N  / 


•  Key  here  is  consistency  and  communication.  Industry  and  the  customer  wilHevolve 
toward  a  common  definition  of  the  services  you  provide.  You  need  to  have  a 
communication  plan  that  is  adaptive  enough  to  adjust  but  managed  enough  that 
there  is  agreement  as  to  context,  content  and  use  of  the  services  you  providedH 
Agreement  is  way  more  important  than  having  the  right  definition.  lt  is  way  too  easy 
to  fall  into  'analysis  paralysis'  trying  to  define  the  essential  services.  It  is  way  better 
to  evolve  a  service  oriented  organization  that  is  always  providing  better  services. 
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W-  Enterprise  Applications  &  Data  Stores: 


LOCKHEED  M  A  R  T  I  N  / 


•  Major  portfolios  aligned  and  individual  initiatives  were  terminated.  Whole 
communities  like  A4/7  (Old  Installations  and  Logistics)  went  on  bare  bones 
sustainment  of  legacy  systems  to  free  funds  for  Expeditionary  Combat  Support 
Systems.  Rigorous  funding  tied  to  effect  producing  capabilities  is  managed  through 
a  hierarchical  portfolio  process  culminating  in  the  Senior  Working  Group  (SWG) 
which  meets  once  a  month  with  the  SecAF  and  reports  status! 
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Another  DoD  Program  Migrating  to  SOA 


Copyright  2007  by  Lockheed  Martin  Corporation 


Goals 


•  Integrated  Portal 

-  Create  an  integrated  Portal  available  to  both  internal  and  external  users 

-  Portal  presents  both  domain-specific  and  multi-domain  JSR  1  SB  compliant  portlets 

*  JSR  168  promotes  capability  of  using  portlets  in  different  Portal  frameworks 

■  Integrated  Data  Access 

-  Develop  single  multi-domain  search  interface  compliant  with  NCE5  Content  Discovery  and  DDMS 

*  Establishes  architecture  for  integration  of  NCES  compliant  data  sources  and  integration  of  DCGS-A  nodes  into 
larger  enterprise  searches 

■  Introduction  of  SOA  Infrastructure  Toolset 

-  Commercial  Enterprise  Service  Bus  (ESB)  used  to  realize  Multi-INT  objectives  though  content- 
based  routing  and  orchestration  of  domain  services  using  BREL  1.1  compliant  workflow 

■  BP  EL  compliance  promotes  capability  of  porting  'workflow  jd  different  workflow  engines 

-  Use  of  UDD1  compliant  service  repository 

*  UDDI  Repository  wrapoed  in  NCES  defined  serv  ices  to  abstract  complexities  of  JDDI  and  promote  portability 
so  different  repository  implementations 

•  Establishment  of  SOA  Governance  Procedures 

-  Government  and  industry  jointly  own  process  for  specification  and  validation  of  service  interface 
standards 

■  Implementation  of  a  Layered  DIB  Compliant  Architecture 

-  User  Facing  Layer  -  Portal  and  Desktop  Visualization  Framework  (MFWS/VIPER) 

-  Processing  Layer  -  Service  Orchestration 

-  Data  Layer  -  Publication  of  Metadata  to  DIB  MDC  for  inter-service  DOGS  interoperability 

-  Core  Layer-  SOA  security  model  based  on  NCES  Security  Services 
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Multi-Tier  Architecture 
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USER  FACING 


14 

EXTERNAL 

USERS 


PORTAL  FRAMEWORT 

CLIENT  FRAMEWORK 

)J 

STAND  ALONE  CLIENTS 

DOMAIN  COMPONENTS 

pOMAIN  ORCHESTRATION 

DOMAIN  WEB  SERVICES 

^  4 

LEGACY  APPS 

CROSS  DOMAIN  COMPONENTS 


* 

EXTERNAL 


> 

CROSS  DOMAIN  ORCHESTRATION 

CROSS  DOMAIN  WEB  SERVICES 

DATA  ACCESS  COMPONENTS 


SOURCES 

METADATA  FRAMEWORK 

h _ 4 

DATA  ACCESS  WEB  SERVICES 

LEGACY  DATA  ACCESS 
COMPONENTS 

k _ 4 

_ 

II 

DATA  BASES  &  INFRASTRUCTURE  SERVICES 


SECURITY 


EXTERNAL 

SYSTEMS 


DATA  MANAGEMENT 
&  INFO  EXCHANGE 


SERVICE  AND  SYSTEM 
MANAGEMENT 
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Enterprise  Search  Capability 


LOCKHEED  M  A  R  T  I  N  / 


SIPRNET  /  JWICS  /  NSANET  (INTERNET) 


/ - \ 


DIB  Compliant 
Enterprise  Search  Engine 


Metadata  catalog 


UDDI  Registry 


DIB  Compliance 
Provides 
Cross-Domain 
and  Cross- 
Service  Capability 
with  Federated 
Enterprise  Query 
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Consolidated  Infrastructure 
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New  Software  Components  Utilize  Common 

Infrastructure 

•  Visualization 

-  Overlay  Manager 

-  Symbol  Manager 

-  NRT  Viz  Stream 

-  Intel  Folders 

•  Data  Management 

-  Metadata  Pub/Sub 

-  Metadata  Discovery 

-  User  Alert  Management 

-  Alert  Services 

-  ISR  Data  Services 

•  Information  Exchange 

-  Information  Gateway 

-  Report  Manager 

-  Message  Logger 

-  Message  Pub/Sub 

-  Address  Book 

Message  Transform 

*  Service  Management 

-  Service  Publish 

-  Service  Inquiry 

SECURITY 

ESB 
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Portal  Framework 


LOCKHEED  M  A  R  T  I  N  / 


•  Portal  Provides  Standard  Visualization  for  Each  Domain 

•  Customizable  to  the  User  Profile 

•  Accessible  from  Any  Workstation  on  the  Network  Including  Laptops) 

•  No  Need  to  Deploy  Client  Applications 
-  New  Capabilities  Added  Quickly 

•  Maintains  Version  Control 

•  Can  Realize  Some  Level  of  Data  Fusion  at  the  Visualization  Layer 
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Portal  Pages 
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Web  Portal  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  lools  Help 


^  Back  **  0^0  0  r\  |  ^  ' Search  Favorites  ^  |  ('Jjr  ^ S 


Address  ]^|  http  ://localhost:  701 1/ 


^  |  Go  Links 


UNCLASSIFIED 


Welcome,  tgreer 


Home  Data  Discovery  Administration  Analyst 


|  Operations 

Geospatial  Geospatial  Management  Tasks  |  Weather  | 

Forecasts  |  Local  Weather  |  Weather  Warnings  Forecast  Products  Weather  Effects 


Local  Weather  Observations  (Advanced) 


Targeting  I  AO  Toolbox  Alerting  Communication  Online  Help 


Winds  f  Visibility  j  Pressure,  j  Temperature  |  Precipitation  j  Lightning  | 
Sensor  METAR  Raw  Data  Graphs 


Current  Winds 
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SmPortletl 
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Ad  vancedSensor  Display  Portlet! 


Winds  j  Visibility  I  Pressure  \  Temperature  if  Precipitation  j  Lightning  j 

Sensor  METAR  Raw  Data  Graphs 


Current  Winds 
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Type  |  Communication  Setup  j  Location  Setup  j  Advanced  | 
Sensor:  |UN KNOWN  jjj 


Operating  Mode 


Live 

Demo  Mode 


I  I 


Mm  ra  Local  intranet 
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ESB  Use 
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•  ESB  used  to  implement  common  service  orchestration  patterns 

-  Content-Based  Routing 

-  Composite  Application 

•  Service  Orchestrations  implemented  to  maximize  portability 

-  BPEL  1.1  used  to  specify  orchestrations  (business  processes) 

-  Use  of  vendor  extensions  strictly  controlled  and  documented 

M  Transformations  into  Program  Specific  Data  Model  done  at  Gateway  and  Data  Manager 
components  to  minimize  dependencies  on  transformation  engines  unique  to  COTS  ESB  variant 

•  Caching  of  UDDI  service  binding  information  used  to  improve  performance 

-  Implement  using  Web  Service  caching  technology 

-  IBM  “ dynacache ”  used  to  cache  NCES  UDDI  Abstraction  Services 

-  Redundant  UDDI  queries  returned  from  cache  as  opposed  to  hitting  the  registry  for  every 
query 


Actions  taken  to  alleviate  SOA  performance  and  portability  concerns  include  pushing 
transformations  down  into  services  and  implementing  web  service  caching  of  UDDI  interfaces 
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Virtual  Machine  Ware  Reduces  HW 
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•  WebLogic  Portal 

WebLogic 

Application  Server 

•  JBOSS  App  Server 

Server 

•23  Web  Apps 

•84  Portlets 

•Six  Web  Apps 

•Sensor  Platform 

•Log4J 

•24+  Web 

Model 

•Windows  2003 

Services 

•Sensor  Platform 

•2  Cores 

•Log4J 

Model  Ul 

•Windows  2003 

•Logger 

•8  GB 

•2  Cores 

•Saba 

•8  GB 

•AAR 

■  m  ’ - n -  If 

n^^03 
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More  Capability  -  Fewer  Vehicles* 
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Mobile  Unit  Transport  Requirements 


CURRENT 

•  14  Vehicles 

•  12  Pull  Behinds 

-  4  Trailers 

-  7  Power  Generators 
- 1  Comms  Unit 

TOTAL  =  26  UNITS 


MIGRATED  CAPABILITY 

•  7  Vehicles 

•  8  Pull  Behinds 

-  2  Trailers 

-  5  Power  Generators 
- 1  Comms  Unit 

TOTAL  =  15  UNITS 
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Integration  and  Test  Improvements 


•  Standards  Compliance  reduces  integration  time 

-  Automated  Test  Tools  Used  to  evaluate  Web  Services  for  OASIS 
Standards  Compliance 

-  Automated  Test  Tools  Used  to  evaluate  Portlets  for  JSR 168 
Compliance 

•  Web  Services  reduces  integration  time  by  encapsulating  the  changes 
to  allow  late  introduction  of  a  capability  or  removing  of  a  planned 
capability  without  incurring  compile  time  errors. 

•  Reduced  regression  testing  when  ipdates  are  made  to  services  that 
do  not  affect  other  services 

•  The  focus  of  the  evaluation  for  accreditation  is  data  centric  instead  of 
system  centric. 
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CONCLUSION 

•  The  initial  investment  in  SOA  infrastructure  can  be  reduced 
through  a  combination  of  mitigation  techniques 

•  Migration  to  SOA  can  increase  capabilities  without 
increasing  the  overall  system  footprint 

•  Initial  investment  in  infrastructure  may  realize  immediate 
operational  benefits 

•  The  anticipated  long  term  beneficidjeffects  of  SOA^gjjgj^l 
migration  are  proving  to  be  realized  in  programs  that  have 
made  the  investment 

•  Multi-Tier  SOA  architecture  canlimit  the  initial  investment 
and  position  a  program  for  long  term  combinatory 
explosion  of  capabilities  as  other  programs  come  online 


* 
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Computer  Modeling  to  Solve  Problems  with 
the  T-38  Propulsion  Modernization  Program 


Presented  on  October  25,  2007 

at  the  NDIA  10th  Annual  Systems 
Engineering  Conference 

by  Randy  Wimer 

USAF  Propulsion  Engineer 

Co-author:  Andy  Hall 

USAF  Propulsion  Engineer 


mm  FORCs 


Briefing  Topics 

Briefing  Objective 
Systems  Engineering 
T-38  Facts 

T-38  Propulsion  Modernization  Program  (PMP) 
Background  and  Hardware  Configuration 

T-38  PMP  -  Three  Issues 

Independent  Review  Team 

Findings: 

-  Issue  #1  -  Engine  Bay  Overheating 

-  Issue  #2  -  Single  Engine  Takeoff  Speed  (SETOS) 
Performance 

-  Issue  #3  -  J85  Engine  Reliability 

Lessons  Learned 
Summary 


Briefing  Objective 


Share  lessons  learned  from  recent  T-38  trainer 
propulsion  modifications 


3.  Interface  Control  Document  (ICD)  should  have  been 
used 

4.  Performance  requirements  should  be  included  as  a 
contract  requirement 


Proper  systems  engineering  -  early  and  throughout 
the  program 

-  Lack  of  computer  (math)  modeling  -  main  tool  in  solving  three 
unrelated  technical  problems _ 

Integration  design  authority  should  be  required  - 
both  government  and  contractors 


Systems  Engineering 

•  Is  an  interdisciplinary  field  of  engineering,  that  focuses  on  the  development 
and  organization  of  complex  artificial  systems 

•  Integrates  other  disciplines  and  speciality  groups  into  a  team  effort,  forming 
structured  development  process  that  proceeds  from  concept  to  production  to 
operation  and  disposal 

•  Considers  both  the  business  and  the  technical  needs  of  all  customers,  with 
the  goal  of  providing  a  quality  product  that  meets  the  user  needs. 


DcvuKi  pmJtnt 
Phasing 


Life 

Planning 


Baselines 


Systems 

Engineering 

Management 


Systems 

Engineering 

Process 


Integrated 

Teaming 


T-38  Facts 

Primary  Function:  Advanced  jet  pilot  trainer 
Builder:  Northrop  Corporation 

Engines:  Two  General  Electric  J85-GE-5  turbojet  engines 
with  afterburners 

Length:  46  feet,  4  inches 

Maximum  Takeoff  Weight:  12,093  pounds 

Maximum  speed:  812  mph  (Mach  1.08  at  sea  level) 

Range:  1,093  miles 

Date  Deployed:  March  1961 

Production  ended:  1972 

USAF  Inventory:  -500  (More  than  1,100  were  delivered 
to  the  US  Air  Force) 

National  Aeronautics  and  Space  Administration  uses  T-38 
aircraft  as  trainers  for  astronauts  and  chase  planes  on 
programs  such  as  the  space  shuttle 


T-38  Propulsion  Modernization  Program 

Background 

•  Managed  by  Ogden  Air  Logistics  Center  (OO-ALC)  at 
Hill  AFB 

•  10  Year  Contract  2001-2010 

•  Engine  upgrade  kit,  inlet,  ejector 

•  Aircraft  modification  at  Randolph  AFB 

-  About  1/3  of  USAF  fleet  modified  when  independent  review  team 
was  formed  in  August  2005 

•  Government  changed  ownership  and  organizational 

structure  3  times  (to  date) 


T-38C  Talon 


PM  and  EN  Authority:  OO-ALC 


PMP  Ejector 

Developed:  GE,  Builder:  GE 
PM  Authority:  OC-ALC 
EN:  OO-ALC 
Installation:  LSI 


PMP  Integration:  OO-ALC 


sTAfe  FORCE 


PMP  J85-5R  Engine 
Developed/Manufactured:  GE 
Engine  PM:  OC-ALC 
Upgrade  Kit  PM:  PRSS  (Contracting) 
EN:  OC-ALC 

Installation:  Laughlin/Columbus  AFB 


PMP  “Fat  Lip”  Inlet  Mod 
Developed:  NASA 
Builder:  CPI 

PM/EN  Authority:  OO-ALC 
Installation:  LSI 


1+ 
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J' 

T] 
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GE-J85-5R  PMP  Engine  Kit 

—  Green  Box  =  USAF  PMP  Sub  Kits 


Spooled  Rotor 

Complete  Set  of  Blades 
Stage  8  Static  Seal 


Combustion  Liner  Assy 

Longer  Life 


Stator  Casing  (AM355) 
Inco  Vanes 
Split-line  Hardware 


Cast  Mainframe 


Turbine  Casing 

Stage  1  &  2  Nozzles, 
Shrouds  & 
Split-line  Hardware 


Ignition  Electrical  System 

Exciter,  Main  &  AB  Plugs  &  Leads 


Kit  1941T47G01 


Improved  Afterburner 

Stacked  Ring  AB  Liner 
VEN  Leaves 
AB  Casing  Finger  Seals 


T-38  PMP  Upgrade  Hangar  at  Randolph  AFB 


T-38  PMP  Issues 


•  Three  issues  arose  during  initial  deployment  of 
PMP  aircraft 


1 .  Engine  bay  overheating 


2.  Single  Engine  Takeoff  Speed  (SETOS) 
Performance 


3.  Modified  engine  did  not  meet  230  hour  Mean 
Time  Between  Removals  (MTBR)  requirement 


Independent  Review  Team 


T-38/J85  IRT  (Independent  Review  Team) 
formed  in  August  2005 

Composed  of  ten  USAF  senior  engineers 

-  Support  from  NASA  and  contractors 

Purpose  was  to  help  solve  technical  issues  with 
T-38  PMP 

Team  held  periodic  reviews  and  site  visits  to  GE 
Hill  AFB,  Laughlin  AFB,  Randolph  AFB,  and 
Edwards  AFB 

IRT  wrapped  up  in  early  2007 


T-38  PMP  Issue  #1-  Engine  Bay 

Overheating 

•  Two  issues:  Cockpit  engine  fire  lights  and 
engine  bay  overheat  events 

-  8  Ground  Occurrences 


-  32  In  Flight  Occurrences  -  all  on  functional  check 
flights  (FCF) 


Engine  Bay  Overheating  Cause 


PMP  introduced  a  new  inlet  and  ejector  that  reduced 
cooling  airflows  in  engine  bays  and  in  keel  cooling 
spaces.  This  decreased  the  tolerance  of  the  system 
to  heat-elevating  conditions. 

Modified  ejector 


T-38A  /  T-38C  Inlet 


Comparison 


T-38A  T-38C  PMP 


From  T-38A  to  T-38C: 

•  +  15%  capture  area 

•  +  5%  throat  area 


T-38  Boat  Tail 


Engine  Bay  with  Boat  Tail  and 
Engines  Removed 


Engine  Bay  Overheating  Findings 

No  Single  Solution,  No  Smoking  Gun 
System  Problem  With  Multiple  Contributions 

-  New  ejector  reduced  engine  bay  cooling  airflow 

-  New  inlet  reduced  keel  cooling  airflow 

-  Hot  air  leaks  around  Variable  Exhaust  Nozzle 
(VEN)  assembly  compounded  problem 

-  Engine  bay  cooling  degraded  over  time  due  to 
old  aircraft  (“tired  iron”)  -  several  leak  paths 

-  Part  to  part  variability 

-  Air  framer  not  initially  involved 


forward 
of  VEN 
housing 


Hot  leaks 


T-38  Boat  Tail  (Old  hardware!) 


Engine  Bay  and  Keel  Space 


Blue  arrows  indicate  keel  airflow 
reversal  under  certain  conditions 

Keel 


Bay 

(tan) 


Engine  Bay  Overheating  Approach 

Ground  test  program  -  collaborative 
NASA/USAF/GE  effort 

-  Aircraft  Configuration  Effects 

-  Engine  Variability  Effects  (VEN  Leaks) 

Flight  test  program  at  Edwards 
Aero-thermo  modeling 


IRT  Engine  Bay  Overheating 
Recommendations 

Validate  thermal  model  in  order  to  determine 
optimum  solution 

-  Operational  solutions  (i.e.  different  FCF  profile)  need 
to  be  evaluated  first  and  design  solutions  second 

Recalibrate  trigger  temperature  for  fire  warning 
system 

Separate  fire  safety  concerns  vs  overheat 
concerns 

Fire  safety  needs  to  be  determined  from 
validated  model  and  design  standards 


T-38  PMP  Issue  #2  -  Single  Engine  Takeoff  Speed 

(SETOS)  Performance 

•  Problem:  Lack  of  confidence  by  operator  that  PMP  performance 
threshold  was  being  met 

•  Operational  Requirements  Document  (ORD)  Requirement  for  hot 
day  takeoff  performance:  Allow  takeoff  (12,5001b  T-38  with  no  wind 
and  1 ,000  ft  pressure  altitude  at  Randolph  AFB)  at: 

8  degrees  F  hotter  (threshold) 

12  degrees  F  hotter  (objective) 


What  is  SETOS? 


SETOS  =  Single  Engine  Takeoff  Speed 
The  faster  of  2-engine  takeoff  speed 

OR 

The  speed  at  which  a  T-38  should  climb  at  100 
feet  per  minute 

-  Out  of  ground  effect 

-  Flaps  extended  60  percent  (takeoff  flaps) 

-  Landing  gear  extended,  gear  doors  closed 

-  One  engine  at  MAX,  the  other  windmilling 


T-38  PMP  Issue  #2  -  Single  Engine  Takeoff  Speed 

(SETOS)  Performance 


Physics: 

-  PMP  inlet  has  improved  inlet  recovery  at  takeoff  conditions 

-  Initial  flight  testing  and  data  analysis  did  not  adequately  verify 
improvement  (inadequate  modeling) 

-  Initial  contractor  model  could  not  be  calibrated 

-  Edwards  AFB  created  a  performance  model  and  conducted 
more  flight  test,  verifying  the  takeoff  thrust  improvement 

-  No  wind  tunnel  testing  done 

-  Measured  thrust  minus  drag  acceleration  in  flight  testing  and 
engine  model  -  used  to  develop  and  validate  the  full  scale 
aircraft  drag  polars 

-  Aircraft  performance  models  included  not  only  engine  net  thrust 
but  also  inlet  recovery,  boat  tail  drag,  inlet  spillage  drag,  and 
secondary  airflow  ram  drag  and  their  engine  power  setting 
dependencies 


T-38  PMP  Issue  #2  -  Single  Engine  Takeoff  Speed 

(SETOS)  Performance 


•  Model: 


-  Updated  USAF  aircraft  performance  model 

-  Confirmed  new  model’s  ability  to  predict  takeoff 
performance 

Flight  test  data  and  analysis  show  SETOS  requirements 
met  or  exceeded 

Findings: 

-  ORD  requirement  to  improve  hot  day  takeoff 
performance  has  been  achieved.  Improved  threshold 
by  1 7  degrees  F 

IRT  also  recommended  that  contractor  create  up  to  date 
aircraft  performance  model  for  T-38C  PMP  aircraft 
configuration 


T-38  PMP  Issue  #3  -  J85  Engine  Reliability 


J85  engine  did  not  meet  230-hour  Mean  Time  Between  Removals 
(MTBR)  Operational  Requirements  Document  (ORD)  threshold 

-  Constant  at  155  hours 

-  PMP  upgrade  focus  was  meeting  maintenance  intervals  and 

shop  cost  reduction 

-  Will  never  get  to  230  hours  without  additional  work  (never  put 

on  contract) 

--  Engine  controls  not  improved 

-  Maintenance  model  did  not  exist 


T-38  PMP  Issue  #3  -  J85  Engine  Reliability 


USAF  did  not  include  MTBR  requirement 

on  GE  contract 

Root  Cause: 

-  PMP  engine  MTBR  currently  same  as  legacy 
engine 

-  Majority  of  removals  are  Unscheduled  Engine 
Removals  (UERs)  -  86%.  Top  UER  drivers 
are  accessories  and  quick  engine  change 
(QEC)  items  (e.  g.  fuel  flow  transmitter) 


Findings  -  J85  Engine  Reliability 


Solution:  A  J85  maintenance  model  was 
Applied  to  the  J85  engine 

Recommendations: 

-  Replace  gear  boxes  with  rebuilt  units  which  have  fuel 
pumps  and  fuel  controls  with  design  improvements 

-  Improve  igniter  system 

-  Implement  ASAP  to  maximize  return  on  investment 

-  Investigate  use  of  contractor  performance  base 
logistics  (PBL) 


Lessons  Learned 


1 .  Proper  systems  engineering  early  and 
throughout  the  program 

2.  Integration  design  authority  should  be 
required 

3.  Interface  Control  Document  should  be  used 

4.  Performance  requirements  should  be 
included  as  a  contract  requirement 


Lessons  Learned 


5.  Use  of  computer  (math)  modeling 

-  Modeling  did  not  exist  for  all  3  issues 

--  Engine  Bay  Flow  -  model  created 
--  SETOS  -  model  used  to  extrapolate 
flight  test  results 

--  Engine  Reliability  -  Existing  model 
applied  to  J85  engine 

-  Airplane  designed  over  40  years  ago 

--  Updated  models  needed 
--  Part  variability 


Summary 


USAF  did  not  use  adequate  systems 
engineering  for  T-38  PMP 

-  Five  major  lessons  learned 

IRT  formed  to  assist  in  solving  three 
technical  issues 

PMP  continuing  to  upgrade  T-38  trainers 
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developing  radar  simulation  models 


Generic  Sensor  Model  (GSM) 


OVERVIEW 

•  Model  Description 

•  Operating  Modes 
o  Stand-Alone 

o  System-of-Systems 

•  Model  Components 

•  Model  Flow 

•  Model  Flexibility 

o  Extensibility 
o  Changeable  Components 
o  System  Adjustable  Parameters 

•  Analyses 


Generic  Sensor  Model  (GSM) 


The  Generic  Sensor  Model  (GSM)  is  a  collection  of 
core  software  components  (classes)  used  as  the 
foundation  for  developing  radar  simulation  models. 

MODEL  UTIUTY 

•  Scenario  Testing 

•  Algorithm  Testing  and  Comparison 

•  Interoperability  Evaluation 

•  Mission  Planning 
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Generic  Sensor  Model  (GSM) 


The  Generic  Sensor  Model  (GSM)  is  a  collection  of 
core  software  components  (classes)  used  as  the 
foundation  for  developing  radar  simulation  models. 

features 

•  Event  Driven 

•  Parameter  Based 

•  Scalable 

•  Modular 

•  Extensible  -  Uses  Object  Oriented  Design  (C++  based) 

•  Can  Incorporate  Tactical  Software 

•  Can  be  Incorporated  into  system-of-systems  environment 
(High  Level  Architecture  (HLA)  interface) 

•  Fidelity  -  configurable  from  low  to  high 
I  Unclassified 
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Generic  Sensor  Model  (GSM) 


The  Generic  Sensor  Model  (GSM)  is  a  collection  of 
core  software  components  (classes)  used  as  the 
foundation  for  developing  radar  simulation  models. 

COMPONENTS 

•  HLA  Interface 

•  Beam  Scheduling 

•  Ray  Trace  Beam  Propagation 

•  Detection  Processing 

•  Tracking 
»  Cueing 

•  Communications 

•  Data  Logging 

•  Dynamic  Environment  (  Atmosphere,  Weather,  clutter ) 

•  Terrain  maps  (DTED) 
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Generic  Sensor  Model  (GSM) 


The  Generic  Sensor  Model  (GSM)  is  a  collection  of 
core  software  components  (classes)  used  as  the 
foundation  for  developing  radar  simulation  models. 

OPERATING  MODES 


Stand-Alone  Mode 

•  All  inputs  via  XML  and  data 
files 

•  All  outputs  to  log  files 

•  Operates  on  a  single 
Windows™-based  platform 


System-of-Systems  Mode 

•  HLA  federated  configuration 

•  Operates  in  Lockheed 
Martin’s  Integrated  Missile 
Defense  Testbed  (IMDT) 
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Generic  Sensor  Model  (GSM) 


Integrate d  Missile  Defense  Testbecj  (I MPT™) 


IMDT  Addresses  All  Phases  of  BMPS  Mission 


1.  Plan  the  Battle 

2.  Fight  the  Battle 

3.  Assess  the  Battle 


-  Integrated  Defense  Planner  (IDP) 

-  IMDT  Federation 

-  Post-Simulation  Analyses 


IMDT  Provides  Aaouruio  BMD  P fanning j 
Porfor/mmoo  And  BynJuntJon  Support 
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Generic  Sensor  Model  (GSM) 


[ntegratecj  Missile  Defense  Testbecj  (IMDT™) 

!MDT_  Federation 

•  Distributed  high-fidelity  system-of-systems 
modeling  and  simulation  testbed  for  BMD 

•  HLA  and  the  GV-Net™  allow  distribution  of  the 
simulation  models  to  their  developers’  (subject 
matter  experts’)  locations 

•  Includes  sensor ,  weapon  systems,  communications, 
and  C2BMC  high-fidelity  models.  System  controller, 
analysis  suite,  and  visualization. 
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Generic  Sensor  Model  (GSM) 


IMDT™  Distributed  Network 


IMDT  Video 
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Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Flow 
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Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Transmit  Event  Flow 


Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Event  Processinq 


Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Components 

•  Physics-Based  Components 
o  Beam  scheduling 

o  Beam  propagating 
o  Signal  calculations 
o  Tracking 

•  Effects-Based  Components 
o  Measured  state 

o  Single-scan  correlation 
o  Multi-scan  correlation 
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Generic  Sensor  Model  (GSM) 

Generic  Sensor  Model  Components 


^ ^ -  - <5 -  - ^ 


Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Components 


Flexibility  and  Extensibility 

•  Beam  Scheduler 
o  Phase  /  Rotate 
o  Phase  /  Phase 
o  Phase  /  Phase  /  Rotate 
o  Track  Filter 

•  Kalman  Filter 
o  Interacting  Multi-Model  (IMM) 
o  Non-Linear  Ballistic  Model 

•  Model  Extensions 

o  External  Cue 
o  IFF 
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Generic  Sensor  Model  (GSM) 

Generic  Sensor  Model  Parameter  Examples 

•  Sensor 

o  Transmitter- power,  duty  cycle, ... 
o  Antenna  -  size,  element  count, ,,, 

•  Waveforms 

o  Selection 

o  Beam  Parameters  -  frequency,  bandwidth, ... 

•  Tracker  Characteristics 

o  Initial  Conditions  -  weights, ... 
o  Operating  Parameters  -  time  constants, 

•  Threats 

o  Number 
o  Characteristics 
o  Trajectories 
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Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Analyses 

•  Component  Performance  Analyses 
o  Detections  -  SNR,  PD,  - 

o  Tracker  -  initiate  track,  drop  track, ... 

•  Algorithm  Analyses 

o  Baseline  updates 
o  Extended  functionality 

•  Mission  Planning 

o  Assumption  verification 
o  Parameter  development 

•  Scenario  Analyses 

o  Targets  -  number,  location,  type, 
o  Assets  -  number,  location,  type, ... 
o  Communications  -  latency,  availability, ... 
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Generic  Sensor  Model  (GSM) 


Generic  Sensor  Model  Analysis  Examples 

•  Stand-Alone  Operating  Mode 

-  Performance  assessment 

■  Track  initiation 

■  Coverage 

■  Detection  probability 

-  Enhanced/Modified  Capability  evaluation 

■  Tracking 

•  System-of-Systems  Operating  Mode 

-  Interoperability 
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Stand-Alone  Mode  performance 
Example 


Individual  missiles  launched 
throughout  the  region  of  interest 

Missiles  impact  one  of  two  cities 
(white,  pink) 

Radar  at  a  specified  location 


Track  initiation  Time 


Track  Duration 


NOTE;  All  chili!  aZ=J  noilonuL 


Track  Coverage 


■  Track  midcourse  only 
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Stand-Alone  Mode  Performance 
Example  Video 


NOTE:  All  doiii  nf'd  noilomiL 
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Stand-Alone  Mode  Enhanced  Capability- 
Example 


ft-  N  Monte-Carlo  runs  using  Tracker  1 

•  N  Monte-Carlo  runs  using  Tracker  2 

•  Evaluate 

-I  Probability  of  track  initiation 
-I  Track  initiation  time 
-I  Track  duration 
-I  Track  drop  time 
-I  Track  quality 
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Interoperability  Video 


NOTE:  All  d'lici  iif'd  noil  os  ml. 


TPS -59  Track  Anti-TEM 


Generic  Sensor  Model  (GSM) 


Summary 

•  Generic  Sensor  Model  (GSM)  provides  a  flexible, 
extensible  framework  for  instantiating  sensor  models 

o  Object-Oriented  design 
o  Parametrically  driven 
o  Stand-Alone  mode 
o  Federated  mode 

•  Integrated  Missile  Defense  Testbed  (IMDT)  provides  a 
distributed  system-of-systems  environment 

o  High-Level  Architecture  (HLA) 
o  Global  Vision  Network  (GV-Net™ ) 
o  Addresses  all  phases  of  the  BMD  mission 

■  Plan  the  Battle 

■  Fight  the  Battle 

■  Assess  the  Battle 
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•  Our  Goal  for  Today 

•  MUOS  in  a  Nutshell 

•  Our  Challenges 

•  Our  Approach  to  Requirements  Management 

•  Lessons  Learned 

•  Challenges  Ahead 
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Why  We’re  Here 


The  history  of  DoD  acquisition  is  littered  with  over-budget, 
late  and  failed  projects 

Why  do  projects  fail  to  stay  within  budget  and  on  schedule? 
Frequently,  the  answer  is  related  to  requirement 

-  Requirements  that  are  unachievable  from  the  start 

-  Poor  management  of  changing  and  emerging  requirements 

-  Lack  of  contractor  oversight  by  the  acquisition  program  office 


•  Today  we  will  discuss  what  the  MUOS  program  has  done  in  an 
attempt  to  avoid  these  pitfalls,  which  we  believe  has  helped  the 
program  stay,  thus  far,  on  budget  and  on  schedule 
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NETWORK  OPERATIONS 


Mobile  User  Objective  System 


•  A  Major  Space  Defense  Acquisition  Program 

•  Tomorrow’s  Narrowband  SATCOM  constellation  to  replace  current 
Ultra  High  Frequency  (UHF)  Follow-On  (UFO)  constellation 

-  Will  provide  lOx  legacy  capacity  and  significantly  improved  availability 

-  On  Orbit  Capability  (OOC)  in  2010 

-  Full  Operational  Capability  (FOC)  in  2014 

•  Critical  Design  Phase  initiated  in  Nov  05/completed  May  07 

-  Contract  for  MUOS  Development  awarded  to  Lockheed  Martin  Team, 
including  Boeing  Satellite  Systems  and  General  Dynamics 

-  Development  of  MUOS  satellites,  ground  systems,  and  user  terminal 
waveform 

•  Key  Decision  Point  (KDP)  -  C  board  completed  Aug  06 
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MUOS  Program  Schedule 


FY  2006 

FY  2007 

FY  2008  FY  2009 

FY  2010 

FY  2011 

FY  2012 

FY  2013 

FY  2014 

1  2  3  4 

1  2  3  4  1 

2  3  4  1  2  3 

4  12  3  4 

12  3 

4 

12  3  4 

1  2  3  4 

12  3  4 

8/1 

10/1 

10/1 

11/7 

7/6 

7/2 

☆ 

☆ 

★ 

☆ 

tT  DDR 

★ 

KDP-C 

Build 

Follow-On 

MRR 

FOC 

Acquisition 

Approval 

Buy  Decision 

3/1 

3/3 

3/3 

3/4 

3/2 

Milestones 

y\ 

A 

A 

A 

A 

A 

><jroc 

^  JROC 

Sat  #1 

Sat  #2 

Sat  #3 

Sat  #4 

Sat  #5 

IPA  □ 

IPA  ES 

00C/I0C 

OOC 

OOC 

OOC 

(Spare) 

OOC 

System  Development 

PDR 

CDR 

- Z-X - 

-A 

1st  and  2nd  satellite,  long  lead-time  items,  entire  ground  infrastructure  and  CAI  development 


Production  Options  for  Spacecrafts  3,  4,  and  5 


DT-B 


DT-C 


Test  &  Evaluation 
Milestones 


_□  DT-D1 
- 1  DT-D2 

t 


DT-D3 


OTRR 

(DT-D2-1) 


ibS^  otrr  ibA( 


OTRR 


OTRR 

TECHEVAL  (DT-D2-2)  TECHEVAL 

OT-D1  OT-D2  OT-D3  OT-D4 

n  □  □  □ 


Phase B 

Preliminary 

Design 


Phase  C 
Complete 
Design 


Phase D 

Build  and  Operations 


1  2  3  4  1  2  3  4  1  2  3  4  1  2  3  4  1  2  3  4  1  2  3  41  2  3  41  2  3  4  1  2  3  4 
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MUOS  Architecture 


On-Orbit  Spare 

-  79°F 


177  W 


_100°W 


15.5  W 


Co-exist  and  operate  with 
Legacy  UFO  in  same  slot 


RAF 


wa 


Radio  Access 
Facilities 

Switching  Facility 
Teleport  l/F 
Network  Mgmt  Facility 
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Our  Challenges 


•  Fixed  Project  Schedule  and  Budget 

-  Aging  legacy  constellation  drives  2010  need  date  for  MUOS  capabilities 

-  Budget  is  always  limited 

•  Sy  ste  m  -of-Sy  ste  m  s 

-  MUOS  is  not  a  stand-alone  system 

-  End-to-end  capability  dependent  on  user  terminals  (JTRS)  and  DoD 
Teleports 

•  Changing  Requirements 

-  MUOS  ORD  has  been  updated  two  times  since  release  of  contract  RFP 

•  Shifting  Strategic  Directions 

-  Introduction  of  the  Net-Ready  KPP 

-  Evolution  of  the  Global  Information  Grid  (GIG) 

-  Movement  toward  packet-switched  technologies 

•  Large  and  Complex  Project  With  a  Distributed  Team 
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MUOS  Requirements  Timeline 


(CY)  2001  2002  2003  2004  2005  2006  2007 

x  7  i  i  i  i  i  i  i 

i  i  i  i  i  i  i 

i  i  i  i  i  i  i 

|  A 

ORD  Approved 

.  .  t  . 

RFP  Released 

ORD  U 

1  A 

pdate1 

l  |  1  A 

POM08  Submission 

POM  10  Subn 

i 

lission 

ORD  Mod 

Component  Advanced  Dev  Phase 


ORD  Update2  CPD  Increment  1 

Risk  Reduction  &  Design  Development  Phase 


1  MUOS  was  grandfathered  from  having  to  convert  to  a  ODD;  this  update  addressed  only 

non-key  performance  parameter  (KPP)  requirements. 

2  This  ORD  update  replaced  the  Interoperability  KPP  with  the  Net-Ready  KPP. 
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Our  Approach  to  Requirements  Mgmt 

The  Big  Picture 


•  MUOS  Program  Management  is  committed  to  controlling  scope 

-  Managing  requirements  is  key  to  controlling  scope,  but... 

-  A  degree  of  flexibility  is  critical 

•  Our  Requirements  Management  goes  well  beyond  simply 
maintaining  documents  and  databases 

-  Our  team  includes  technical  experts  to  manage  the  content  of  the 
requirements 

-  We  utilize  and  customize  tools  to  provide  the  program  office 
comprehensive  insight  and  oversight  of  requirements 

•  Requirements  are  being  managed  throughout  the  program’s  entire 
life  cycle 
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Our  Approach 

Unachievable  Requirements. . . 


(CY)  2001  2002 

- T — *" 

ORD  Approved 


2003 

f  T 

ORD  Mod 

Component  Advanced  Dev  Phase 


•  We  participated  in  all  early  operational  requirements  meetings  with 
warfighter  representatives  (COCOMS,  services,  OSD,  etc.) 

-  Raised  issues  with  requirements  that  were  deemed  to  push  the  limits  of 
technology  at  the  time,  which  influenced  the  initial  operational  requirements 
(ORD) 

•  CAD  phase  was  an  opportunity  for  us  to  further  refine  the  requirements 
with  2  contractor  teams 

-  Provided  insight  into  how  the  requirements  were  driving  the  architectures 

•  As  potential  architectures  matured,  we  worked  with  the  resource  sponsor 
and  user  community  to  address  requirements  that  were  significantly 
impacting  architecture  designs  and  costs 

-  Successfully  reduced  2  requirements  (1  a  key  performance  parameter)  as  a 
result  of  both  cost  as  an  independent  variable  (CAIV)  and  technical  feasibility 
studies 
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Our  Approach 

Changing  and  Emerging  Requirements... 


(CY) 


2004 


2005 


2006 


2007 


TT 

}ased  I 


RFP  Released 

ORD  Update 


7 


POM08  Submission 


T 


POM  10  Submission 

I 

ORD  Update  CPD  Increment  1 

Risk  Reduction  &  Design  Development  Phase 


•  MUOS  has  successfully  managed  2  post-RFP  ORD  updates 

-  Contractor  performed  impact  assessment  of  new  &  changed  requirements 

-  Results  allow  us  to  make  informed  budget,  schedule,  performance  and  risk 
decisions  and  elevate  issues 

•  We  are  allocating  requirements  to  2  Increments 

-  Current  capability  is  documented  in  the  MUOS  CPD  Increment  1 

-  Remaining  requirements,  when  funded  and  contracted,  will  go  into  CPD 
Increment  2 

•  Requesting  funds  through  resource  sponsor  via  POM  inputs 

•  We  track  applicable  policies  and  DoD  directives  and  work  closely  with 
NSA... 

-  incorporating  derived  requirements,  when  possible 

-  raising  issues,  as  necessary 
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Our  Approach 

Changing  and  Emerging  Requirements... 


2005  20Q6  2007 

~ 1  1  r 

POM  10  Submission 


Risk  Reduction  &  Design  Development  Phase 


•  We  acknowledge  we  are  part  of  a  system-of-systems 

-  We  work  hard  to  remain  aware  of  JTRS  and  Teleport  architecture 
development  and  changes  and  address  issues  as  early  as  possible 

•  Established  the  MUOS  Acquisition  Council 

•  Active  Interface  Control  Working  Groups 

-  We  stay  ahead  of  Net-Centric  directives  to  minimize  impacts  while  making 
smart  decisions  -  i.e.,  evolving  with  our  environment  within  our  constraints 

•  Example  is  early  decision  to  change  to  an  entirely  packet-switched  architecture 

•  Moving  ahead,  we  continue  to  communicate  with  the  user  community  on 
emerging  requirements 

-  Conducted  meeting  with  warfighter  reps  to  understand  their  priorities,  which 
will  influence  POMIO  submission  and  future  allocation  of  funds 
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Our  Approach 

Contractor  oversight  by  the  program  office... 


Component  Advanced  Dev 

Phase  Risk  Reduction  &  Design  Development  Phase 


•  We  translated  the  operational  requirements  into  more  detailed  and 
testable  requirements  via  the  contract  performance  specification 

-  Frequently  requested  clarifications  of  the  requirements  by  participating  in 
every  ORD  meeting,  email,  phone  calls,  etc. 

-  Involved  subject  matter  experts  across  the  program  office  team 

•  We  have  a  very  productive  relationship  with  STRATCOM’s  MUOS 
representative,  ensuring  we  consider  the  user’s  perspective 

-  He  frequently  participates  in  IPT  meetings  and  attends  design  conferences 

•  We  continually  foster  frank  and  open  communications  with  the  contractor 

-  Contractor  not  only  requests  clarification  of  requirements,  but  also. . . 

-  Contractor  has  proposed  refinements  to  requirements  to  allow  improved 
design  and/or  cost  savings 
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Our  Approach 

Contractor  oversight  by  the  program  office... 


(CY) 


2005  20Q6 


2007 


Risk  Reduction  &  Design  Development  Phase 


•  We  work  very  closely  with  the  developer  to  ensure  proper  treatment  of 
contract  requirements 

-  Tracking  requirements  flow  and  interpretation  through  all  levels  of 
specification:  system,  segment,  subsystem,  software  and  hardware 

•  Are  contract  specifications  fully  captured? 

•  Have  extraneous  requirements  been  added? 

•  What  are  the  implications  of  a  high-level  requirements  change  or  a  low- 
level  design  change? 

-  Strong  participation  in  and  oversight  of  requirement  verification  plan 
development 

•  Tools  are  key  to  our  processes 
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Requirements  Management  Tool 

Utilize  and  Customize 


Government 
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|  IFGPxx  1  1  IA  | 

4  SIM  ]  I— Multiple 


•  Large  Program 

-  160  specifications 

-  500,000  rqmts 

•  Distributed  Team 

-  3  DOORS  databases 

-  weekly  integrations 

-►  21,000  lines  of  code 


Tools  are  a  must! 


Mng  Rqrmts  Brief 
PMW-146-D-07-0184 


15 


UNCLASSIFIED 


Requirements  Management  Tool 

Involve  the  Requirements  Managers 


Requirement  Managers 

•  active  IPT  members 

•  help  the  IPTs  use  the  tools 


Collaborate  to  use  the  tools 
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Requirements  Management  Tool 

Customized  Reports 
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Pivot  tables 


Customized  reports  have  become  a  central  aspect  to  the  daily 

management  of  MUOS 
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Requirements  Management  Tool 

Customize  the  Requirements  Database 


OPTEVFOR 


Expanded  existing  tools  to  address  changing  Program  office  needs 
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Lessons  Learned 


•  The  Positive 

-  Program  management  must  be  committed  to  managing  project  scope, 
yet  have  enough  flexibility  to  respond  to  the  changing  environment 

•  Examples:  change  to  IP  technology,  evolving  interfaces  with  DISN  services 

-  Rigorous  interaction  with  the  requirements  community  is  crucial 

•  Manage  expectations  by  ensuring  requirements  are  feasible,  affordable 
and  understood 

-  Meticulous  oversight  of  top  to  bottom  requirements  is  a  necessity 

•  Tools  are  essential  here:  requirements  traces,  verification  pivot  tables,  etc. 

-  Invest  in  a  robust  Requirements  Management  System 

-  Foster  a  cooperative  environment  that  is  receptive  to  change 

•  Cooperation  among  SETA  contractors  significantly  enhances  productivity 

•  Development  contractor  suggestions  have  resulted  in  many  improvements 

•  Needs  Improvement 

-  Escalation  of  process  problems 

•  At  times  we  wasted  months  trying  to  resolve  process  issues  with  the 
developer 

-  Coordination  with  other  projects 

Mng  Rqrmts  Brief  •  No  overarching  integrator 

PMW-146-D-07-0184 


19 


UNCLASSIFIED 


Challenges  Ahead 


•  Broader  systems  engineering  issues  to  keep  our  large,  highly 
complex  project  on  budget  and  schedule 

-  We’re  already  facing  issues  that  are  impacting  budget  and  schedule 

•  Technical  issues 

•  Security  issues 

•  Continued  reliance  on  other  projects 

-  User  terminal  developments  are  well  behind  MUOS  development 

-  Teleport  funding  is  out  of  sync  with  MUOS  development 

-  Lack  of  an  overarching  integrator  increases  this  challenge 

•  Dealing  with  future  new  requirements 

-  New  requirements  have  been  proposed 

-  We  are  currently  working  through  a  process  to  address  these, 
potentially  developing  a  CDD  for  a  block  upgrade 
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You  can’t  always  get  what  you  want . . . 


You  can’t  always  get  what  you  want, 
But  if  you  try  sometimes,  well,  you  just  might  find, 


You  get  what  you  need! 

(M.  Jagger/K.  Richards:  1968) 
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What  Does  a  Program  Manager  Want ? 


A  high  quality  product  delivered 
within  cost  and  schedule. 
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A  Program  Manager  Also  Needs... 


Accurate,  high-quality  data 


and  information  to  keep  the  program  or 


project  on-time  and  on-budget. 
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Topics 


Key  Systems  Development  and  Program 
Management  Issues 

Metrics  for  Managing  Cost  and  Schedule  Issues 
Project  Planning  Principles  and  Application 
Project  Tracking  Principles  and  Application 
The  Way  Forward  .  .  .  Team  Process  Integration 
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Key  Issues  in  Managing  Projects/Programs  - 1 


Management  may  not  be  used  to 
working  with  data  . . . 


Management  doesn’t  know 
what  to  ask  for . . . 
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Key  Issues  in  Managing  Projects/Programs  -  2 


Different  levels  of  management  &  needs  for  data/information 

•  Frequency  and  type 

•  Project  I 

•  cross-project  reviews 

•  Quarterly 

•  program/milestone 

•  Organizational  Level 

•  First  level  management:  team  level 

•  Second  level:  middle  management — whoever  the  team  reports  to; 
responsible  for  half  dozen  projects;  matrix  management  issues 

•  Third  level:  organization  level,  business  managers,  people  who  manage 
the  budget  and  people  who  manage  the  process 


B 
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Key  Issues  in  Managing  Projects/Programs  -  3 


Acquisition 

•  What  did  you  ask  for  in  the  RFP  or  task  order? 

•  Contractor  and  sub  contractor  relationships  &  reporting 


Data  analysis,  summary  &  presentation 

•  Dashboard/scorecard 

•  Stoplight  charts 

•  Better  (e.g.,  PQI,  PDF,  defect  densities,  customer  satisfaction  .  .  .) 

•  Faster  (e.g.,  test  time  reduction,  rework  time,  ratio  of  development  time  to 
total  time,  ...) 

•  Cheaper  (e.g.,  cost  in  FTE  months  or  $$  to  produce  1 ,000  task  hours  of 
work,  ...) 
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What  Data  Should  be  Collected,  Analyzed,  and 


Primarily  status/performance  data  and 
information: 

•  Where  are  you  in  your 
program/project? 

•  How  do  you  know? 

•  Show  me! 

•  EARLY  WARNING! 


Project/Program  Performance 

•  Schedule 

•  Quality 

•  Cost 

•  Features 

•  Risks 
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Other  Questions  . . . 


What  is  the  availability  of  resources  for  a  particular  project? 
What  is  the  productivity? 

How  good  is  my  organization? 

What  is  the  estimation  accuracy? 

What  is  the  cost  to  develop  x,  y,  z? 

How  large  are  our  products? 

How  are  we  performing  (size,  effort,  schedule,  quality,  ...)? 
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All  Projects  Have  Cost  and  Schedule  Issues 


•  Most  projects  are  underestimated,  usually  by  a  lot,  only  few 
are  close. 

•  Many  project  plans  are  missing  an  unknown,  some 
unknowns  are  disasters,  and  all  will  have  impact. 

•  Many  projects  are  late  before  they  start,  some  know  it,  and 
some  don’t. 

All  projects  have  issues.  The  question  is: 

“How  will  you  manage  the  trouble  you’re  in?” 
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Cost  and  Schedule  Trouble 


When  the  program  manager  wants  to  know  .  .  . 

“What  is  the  status  of  your  project?” 

“How  do  you  know?” 

“Show  me!” 

.  .  .  How  will  you  respond? 

It’s  a  question  of  how  to 

•  identify  the  real  issues  and  alternatives 

•  define  a  realistic  course  of  action 

•  take  it 
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Topics 


Key  Systems  Development  and  Program 
Management  Issues 

Metrics  for  Managing  Cost  and  Schedule  Issues 
Project  Planning  Principles  and  Application 
Project  Tracking  Principles  and  Application 
The  Way  Forward  .  .  .  Team  Process  Integration 
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Conventional  Wisdom 


Advice  is  plentiful  and  full  of  good  intent.  So  what’s  wrong  with  the 
following  guidelines? 


Program  Management  Critical  Success  Factors 

Cost  and  schedule  estimates  are  realistic  and  current. 

Demonstrable  Indicators 

Possible  Sources  of  Evidence 

Does  the  scope  of  the  estimate  cover  all 
activities  throughout  the  system  life? 

Costed  Through-Life  Management 

Plan. 

Were  independent  methods  used  to  produce 
and  check  cost  and  schedule  estimates? 

Estimates  available  for  reviewer. 

Have  the  estimates  been  independently 
validated? 

Independent  Technical  Review,  Quality 
Assurance  Review. 

Are  there  mechanisms  to  monitor  and  adjust 
assumptions  and  estimates? 

Process  documentation  /  Quality 

System 

^=-  Software  Engineering  Institute 
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Partial  List  of  Answers  . . . 


•  Lagging  indicators 

•  Metrics  disassociated  from  work  and  those  performing  it 

•  Latency  in  reporting 

•  Mechanism  to  adjust  estimates  and  replan  over  the  life  of  the 
project? 

•  How  can  we  do  better  next  time? 
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SEI  Core  Metrics 


Size 


Quality 


Software  Engineering 


Effort 


Schedule 


Carnegie  Mellon 
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What  PM’s  Should  Expect 


Estimated  and  Actual  Size 


Size  is  a  measure  of  the  magnitude  of  the 
deliverable 

•  for  software:  lines  of  code  or  function 
points 

•  for  other  intellectual  work:  document 
pages,  presentation  slides,  computer 
tests  or  simulations,  etc. 

Size  is  estimated  and  reported  for  each 
component. 

The  actual  size  is  measured  and  reported 
for  each  component. 

Size  data  are  used  to 

•  estimate  effort 

•  normalize  other  measures 

•  track  progress 


Intellectual  work  can  be  measured 
according  to  accounting  categories. 


•  Base  <measure> 

•  Reused  <measure> 

•  Added  <measure> 

•  Modified  <measure> 

•  Deleted  <measure> 
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What  PM’s  Should  Expect 


Estimated  and  Actual  Effort 


Effort  is  a  measure  of  time  “on 
task.” 


The  recommended  effort 
measure  is  a  task  hour. 


Task  hours  should  be 
measured  and  reported  by 

•  WBS  task 

•  process  phase 

•  project  week 


How  many  task  hours  are 
there  in  a  40  hour  week? 

About  15  to  20! 
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What  PM’s  Should  Expect 


Estimated  and  Actual  Schedule 


Schedule  shows  the  work 
completed  or  planned  to  be 
completed  in  a  given  period. 

Schedule  accuracy  depends  on 
granularity  of  tasks. 

Recommended  task  granularity 

•  From  1 0  to  20  hours  for  near- 
term  tasks 

•  From  two  to  four  weeks  for 
remaining  tasks. 

Estimated  and  actual  task  dates  are 
reported. 
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What  Acquirers  Should  Expect 


Estimated  and  Actual  Quality 


Defects  are  a  measure  of  quality. 


Definition:  when  the  product  must  be 
changed  to  ensure  proper  design, 
implementation,  test,  use,  or  maintenance,  a 
defect  has  been  identified. 


Estimates  of  the  number  of  defects  injected 
and  removed  in  each  phase  and  component 
should  be  reported. 


Actual  defects  found  in  each  component, 
and  each  phase,  and  the  phase  in  which  the 
defect  was  injected  should  be  reported. 
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Derived  Measures 


The  core  metrics  support  many  derived  measures  that  can  be  used  by 
the  acquirer  to  evaluate  plans  and  track  program  status. 

Examples  include 


planned  value,  earned  value, 
and  predicted  earned  value 

estimation  accuracy  for  size, 
time,  and  quality 

time  in  phase  distribution 

defect  density 

defect  removal  rate 


review  rates 

process  and  phase  yield 

cost  of  quality 

percent  defect-free 

defect  removal  profiles 

quality  profile  and  process 
quality  index 
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Topics 


Key  Systems  Development  and  Program 
Management  Issues 

Metrics  for  Managing  Cost  and  Schedule  Issues 
Project  Planning  Principles  and  Application 
Project  Tracking  Principles  and  Application 
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Project  Plans 


A  Program  Manager  should  expect  to  find  the  following: 

•  Most  project  plans  are  underestimated. 

•  Many  project  plans  are  missing  the  unknowns. 

•  Many  projects  are  late  before  they  start. 

The  problem  is  that  many  project 


plans  ignore  well-known  principles 
of  project  planning  and  are 
therefore  un-executable. 


Project  Planning  Guidelines 


Projects  are  dynamic,  so  are  good  project  plans. 

Accuracy  improves  with  detail,  so  make  detailed  plans  and 
estimates;  use  historical  data. 

Quality  must  be  managed  by  the  numbers,  so  make  a 
quantitative  quality  plan. 

The  best  plans  are  made  by  the  people  assigned  to  do  the 
work. 
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Dynamic  Plans 


If  you  can’t  make  accurate  plans,  plan  often. 

Many  projects  (especially  software  projects)  are  subject  to 
considerable  change. 

Every  change  will  affect  the  cost  and  schedule. 

Every  cycle  will  include  new  information. 

Projects  should  produce  new  plans  every  few  n 

•  a  detailed  near-term  plan 

•  a  high-level  plan  to  the  end  of  the  project 
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Detailed  Plans 


Detailed  plans  are  more  accurate. 

They  are  more  likely  to  be  complete. 

There  is  greater  opportunity  for  over-estimates  and  under¬ 
estimates  to  cancel. 

Detailed  plans  provide 

•  deeper  insight  into  the  work 

•  better  guidance  during  execution 

•  more  accurate  tracking 

•  basis  for  getting  out  of  trouble 
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Detailed  Estimates 


Like  detailed  plans,  detailed  estimates  are  more  accurate  for 
the  same  reasons. 

There  are  many  estimating  methods  available. 

The  best  methods 

•  use  historical  data 

•  have  a  feedback  loop 

•  are  systematic  methods  or  processes  that  reduce  bias  and  can  be 
improved 

•  are  appropriate  for,  and  calibrated  to,  the  work 
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Estimating  Bias 


Most  estimates  tend  to  be  optimistic. 

The  key  to  accurate  estimates  is  not  the  accuracy  of  each 
estimate. 

The  key  to  accurate  estimates  is  detail  and  consistency. 

To  improve  consistency,  the  engineer  must  reduce  estimating 
bias. 

To  reduce  estimating  bias  the  developer  should 

•  make  size  estimates  first 

•  base  effort  on  size 

•  next  estimate  resources 

•  use  estimated  effort  and  resources  to  create  a  schedule 
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Quality  Principles 


Many  intellectual  products  ignore  quality  until  final  review  (or 
test). 

You  can’t  test  in  quality;  if  you  try  to  you  will 

•  increase  cost  and  schedule  by  40%  to  50% 

•  make  the  plan  even  more  unpredictable 

•  pay  the  contractor  for  avoidable  rework 

•  field  a  defective  product 

To  get  a  quality  product  out  of  test,  you  must  put  a  quality 
product  into  test. 

To  put  a  quality  product  into  test  you  must  measure  and 
manage  quality  at  each  step  of  the  process. 
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Quantitative  Quality  Plans 


Quality  without  numbers  is  just  talk. 

If  you  want  a  high-quality  product  you  must  ask  for  quality 
data. 

To  do  this  you  need  to  know 

•  what  quality  data  to  ask  for 

•  how  to  evaluate  the  quality  data  you're  given 

The  core  metrics  are  a  good  start. 

•  defects  injected  and  removed 

•  by  phase  and  component 

•  both  estimated  and  actual 
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Quality  Benchmarks  (software  example) 


These  quality  benchmarks  are  derived 
from  the  SEI’s  TSP  research  for 
software  projects. 


They  are  based  on  results  obtained  from 
software  project  teams  and  individual 
developers. 


They  can  be  adapted  to  evaluate  or 
track  other  kinds  of  project  plans. 


Sample  TSP  Benchmark  Data 


Measure 

Benchmark 

Review  rate 

200  LOC/Hr. 

Code  injection  rate 

2/Hr. 

Code  review  removal 
rate 

4/Hr. 

Code  inspection 
removal  rate 

1/Hr. 

Code  review  and 
inspection  yield 

70% 

Unit  test  removal  rate 

3/Hr. 

Unit  test  yield 

50% 

Integration  and 

System  Test  removal 
rate 

>10  Hr.  per  defect 

Integration  and 

System  Test  yield 

<  50% 
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Evaluating  Plans  -1 


The  objective  is  to  determine  if  the  plan  represents  a 
reasonable  commitment. 

Is  it  complete,  detailed,  accurate,  robust,  and  capable  of 
producing  a  quality  product? 

Size  estimate 

•  Was  size  estimated? 

•  How  were  the  estimates  made? 

•  Are  the  size  estimates  detailed? 

•  Was  historical  data  used? 

•  Who  made  the  size  estimates? 
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Evaluating  Plans  -2 


Effort  estimate 

•  Was  effort  estimated? 

•  How  were  the  estimates  made? 

•  Are  the  efforts  estimates  based  on  size? 

•  Are  the  efforts  estimates  detailed? 

•  Was  historical  data  used? 

•  Who  made  the  effort  estimates? 

Quality  plan 

•  Does  the  plan  include  estimates  of  defects  injected  and  removed  by  phase? 

•  How  were  these  estimates  made? 

•  Are  they  based  on  either  size  or  effort? 

•  Was  historical  data  used? 

•  Who  made  the  estimates? 

•  How  is  the  effort  distributed?  (gets  to  quality  and  rework) 
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What  Acquirers  Should  Look  For 


Probing  Questions  for  a  Plan  Review 


1 .  What  are  the  size  estimates  for  the  principal 
products  and  how  did  you  arrive  at  these  estimates? 


2.  How  do  you  know  if  you  have  included  all  of  the 
necessary  tasks? 


3.  What  is  the  basis  for  your  task  estimates? 


4.  How  did  the  team  arrive  at  the  schedule? 


5.  How  did  you  estimate  the  number  of  weekly  task 
hours? 


6.  What  could  management  do  to  help  maximize 
your  weekly  task  hours? 


7.  Have  you  looked  at  alternative  plans? 


8.  Could  you  accelerate  the  schedule,  and  if  so, 
what  would  it  cost? 


9.  What  is  your  quality  plan  and  how  did  you 
produce  it? 


10.  What  is  your  security  plan  and  how  did  you 
produce  it? 


1 1 .  What  are  the  key  risks  and  your  mitigation  plans 
for  them? 


12.  Have  you  considered  prototyping  or  reuse? 


13.  What  is  the  likely  error  in  the  size,  time,  and 
schedule  estimates? 


14.  When  will  you  have  learned  enough  to  make  a 
more  accurate  plan? 
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Commitment 


Intellectual  product  development  is  a  team  sport. 

Motivated  teams  are  more  productive  and  produce  better 
products. 

When  teams  are  empowered  to  make  their  own  plans  they  are 
motivated  to  meet  their  commitments. 

Costs  and  schedules  are  established  before  the  project  is 
staffed  and  are  difficult  to  change. 

The  question  is,  how  early  do  you  want  to  know  about  cost 
and  schedule  trouble? 
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Getting  Out  of  Trouble 


Projects  don’t  get  behind  overnight,  they  slip  day  by  day. 

If  you  manage  the  days,  the  weeks,  months,  and  years  will 
take  care  of  themselves. 

The  key  is  early  warning. 

•  The  review  period  must  be  short. 

•  Detailed  weekly  data  are  needed. 

The  most  common  causes  of  schedule  slip 

•  estimation  error  (size,  effort,  hours  available  per  week) 

•  quality  problems 
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Early  Warning 


Projects  get  in  trouble  for  many  reasons. 


What  indicators  provide  early  warning  of  cost,  schedule,  or 
quality  problems? 

•  earned  value  progress  tracking 

•  resource  tracking 

•  software 

—  defect  removal  profile 

—  percent  defect-free 
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Earned  Value  Progress  Tracking 


The  SEI  recommends  the  use  of  earned  value  for  tracking 
project  progress. 

Earned  value  (EV)  is  a  method  for  tracking  progress  based  on 
the  contribution  that  each  task  makes  towards  completing  the 
project. 

EV  provides  a  precise  way  to 

•  measure  project  status 

•  estimate  project  completion  dates 
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Earned  Value  Guidelines 


The  SEI  recommends  the  use  of  the  following  earned  value 
guidelines. 

•  A  task’s  EV  is  the  task’s  estimated  hours  divided  by  the  total 
estimated  hours  for  all  tasks  in  the  project. 

•  The  project  earns  the  value  for  a  task  when  it  is  completed,  there  is 
no  partial  credit. 

•  The  EV  credit  is  the  same,  regardless  of  the  time  the  task  actually 
takes. 
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Earned  Value  Tracking  (toward  a  miles*"- 


100.0 


Cumulative  planned 
value  shows  the 
current  plan. 


Baseline  cumulative 
planned  value  shows 
the  initial  plan. 


Using  the  rate  of 


progress  as  a  basis, 
predicted  earned 
value  shows  the  likely 
completion  date. 


Week 
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EV  Tracking  Considerations 


Causes  of  EV  slip 

•  difference  in  estimated  vs.  actual  effort 

•  difference  plan  vs.  actual  resource  hours 

•  blocked  tasks 

EV  limitations 

•  assumes  future  is  like  the  past 

•  insensitive  to  future  estimation  error  or  resource  deltas 

•  poor  quality  in  test  can  be  a  significant  impact 

EV  must  be  applied  consistently  across  projects  and  programs! 
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Status  Tracking  at  a  granular  level 


Here’s  an  example  of  how  teams  can  track  their  status  weekly  using  a 
defined  process  and  a  weekly  status  summary  in  a  support  tool. 


TSP  Week  Summary  -  Form  WEEK 

Name 

Team 

Status  for  Week 

Week  Date 

Carol 

Weekly  Data 

Schedule  hours  forthis  week 
Schedule  hours  this  cycle  to  date 
Earned  value  forthis  week 

Earned  value  this  cycle  to  date 
To-date  hours  fortasks  completed 
To-date  average  hours  perweek 

Date 

Cycle 

Plan 

151.0 

4/7/2003 

Plan/ 

Actual 

1.76 

PSP  Ghost 

IS" 

7j 

3/10/2003 

Actual 

86.0 

1526.0 

1594.8 

0.96 

6.9 

4.2 

1.64 

79.5 

84.3 

0.94 

1580.7 

1568.1 

1.01 

101.7 

106.3 

0.96 

Assembly 

Phase 

Tasks  Completed  or  Due 

Resource 

Task  Plan 

Hrs. 

Task 

Actual  Hrs. 

Earned  or 

Plan  Value 

Planned 

Week 

Plan  vs. 

Actual  Hrs. 

Main  Form 

CODEINSP 

Main  Form  Code  Inspection 

SA 

1.5 

2.4 

0.1 

10 

0.63 

OEMMOO  Delivery.aspx 

UT 

OEMMOO  Delivery.aspx  (FE-Server)  L 

JNK 

8.9 

3.0 

0.5 

13 

2.91 

OEMMOO  Deliveiy.aspx 

DLDINSP 

OEMMOO  Delivery.aspx  (FE-Client)  D 

INK 

0.0 

0.0 

0.0 

13 

DEMMOO  Delivery  asp^ 

LU 

Q 

o 

O 

OEMMOO  Delivery.aspx  (FE-Client)  C 

iNK 

7.5 

5.7 

0.4 

14 

1.32 

DEMMOO  Delivery.aspx 

OR 

OEMMOO  Delivery.aspx  (FE-Client)  C 

iNK 

3.8 

1.7 

0.2 

14 

2.26 

DEMMOO  Delivery.aspx 

COMPILE 

OEMMOO  Delivery.aspx  (FE-Client)  C 

iNK 

1.3 

0.9 

0.1 

14 

1.44 

DEMMOO  Deliveiy.aspx 

CODEINSP 

OEMMOO  Delivery.aspx  (FE-Client)  C 

iNK 

0.0 

0.0 

0.0 

14 

DEMMOO  Delivery.aspx 

UT 

OEMMOO  Delivery.aspx  (FE-Client)  U 

INK 

5.9 

6.8 

0.3 

14 

0.87 

Query  Object 

TD 

Query  ObjectTest  Development 

MB 

0.0 

0.0 

0.0 

14 

Query  Object 

CODEINSP 

Query  Object  Code  Inspection 

MB 

0.0 

1.2 

0.0 

14 

0.00 

■"In am  Hhior-t 

1  IT 

Ciuant  Pihior-t  1  In  it  To  ct  Hi-alnnc 

twin 

i  i 

1  7 

n  i 

1  A 

n  rk 
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Resource  Management 


Here’s  another  example  of  how  teams  can  track  resource  utilization  weekly 
using  a  defined  process  and  a  weekly  status  summary  in  a  support  tool. 


1200.0 

1000.0 

800.0 

600.0 

400.0 

200.0 

0.0 


Cumulative 

Planned 

Hours 

Cumulative 
Actual  Hours 


Week 
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Quality  Management 


Here’s  another  example  of  how  Software  Development  teams  can  use  a 
Quality  Profile  as  an  early  warning  indicator  of  post-development  defects. 

The  quality  profile  uses  five  software  quality  benchmarks. 

Satisfied  criteria  are  plotted  at  the  outside  edge  of  the  chart. 


Inadequate 
design  review 
time  results  in 
design  defects 
escaping  to 
test  and 
production. 


Component  5  Risk  Factors 


Design/ Code  Time 


Design  Review  Time 


Unit  Test  D/KLOC 


Code  Review  Time 


Compile  D/KLOC 
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Defect  Removal  Profile 


Again,  using  a  Software  Development  example,  a  Defect  Removal  Profile 
may  be  used  to  track 

•  plan  and  actual  defects  removed  by  phase 

•  early  vs.  late  defect  removal  plan 


Percent  Defect  Free 


Similarly,  teams  can  use  a  Percent  Defect-Free  profile  to  track 
•  percentage  of  components  that  are  defect-free  in  test 


•  pre-test  process  quality 
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What  Acquirers  Should  Look  For 


Probing  Questions  for  a  Status  Review 


1 .  What  percentage  of  the  project  has  been 
completed  based  on  earned  value? 


2.  How  does  the  earned  value  compare  to  planned 
value? 


3.  What  is  the  end  date  based  on  predicted  earned 
value? 


4.  How  do  planned  resource  hours  compare  to 
actual  resource  hours? 


5.  How  do  actual  hours  for  completed  tasks 
compare  to  estimated  hours  for  completed  tasks? 


6.  What  is  the  gap  between  actual  resource  hours 
and  actual  hours  for  completed  tasks? 


7.  How  does  this  gap  compare  to  the  average  actual 
hours  per  week? 


8.  How  do  size  estimates  compare  to  the  actual  size 
of  completed  components? 


9.  What  is  the  correlation  between  actual  size  and 
actual  hours  for  completed  components? 


10.  What  is  the  correlation  between  the  planned 
size  and  actual  hours  for  completed  components? 


1 1 .  How  do  planned  defects  compare  to  actual 
defects  for  completed  phases  and  components? 


12.  What  is  the  actual  review  rate  for  inspections 
and  reviews? 


13.  What  percentage  of  components  are  defect-free 
at  unit  test,  integration  test,  system  test,  ...? 


14.  How  does  the  actual  defect  removal  profile 
compare  to  plan  for  completed  components? 
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What  Should  Be  the  Way  Forward? 


Alternatives  should  address  deficiencies  in  “conventional 
wisdom” 

•  Based  on  data  collected  as  work  is  done 

•  Used  to  change  development  behavior 

•  Predictive  metrics,  not  just  reactive  (leading  vs.  lagging) 

•  Using  proven,  repeatable  methods 

•  Incorporating  planning  and  tracking  principles  we  have  just 
discussed 

•  Addressing  institutionalizing  change 

If  we  don’t  address  these  issues,  we  are  doomed  to  repeat  the 
mistakes  of  the  past .  . . 
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What  Alternatives  Exist? 


One  example  is  the  Team  Software  Process  (TSP),  an  integrated  set  of 
practices  for  developing  software. 

TSP  is  an  process-based  solution  to  common  software  engineering  and 
management  issues. 

•  cost  and  schedule  predictability 

•  productivity  and  product  quality 

•  process  improvement 
Unlike  other  methods,  TSP 

•  teams  are  self-directed. 

•  emphasizes  measurement  and  quality  management. 

•  provides  immediate  and  measurable  benefits. 

•  accelerates  CMMI-based  improvement. 


Software  Engineering  Institute  CarnegieMellon 


How  To  Talk  To  A  Program  Manager 

©  2007  Carnegie  Mellon  University 


51 


TSP  Improves  Performance 


Average  Effort  Deviation  -  Range 

120%  n— 

100% 

80% 

60% 

40% 

20%  _ 

0% 

-20%  J— 

Pre  TSP/PSP  With  TSP/PSP 


Average  Schedule  Deviation  -  Range 


Defects/KLOC  in  Acceptance  Test  -  Range 


0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 

Pre  TSP/PSP  With  TSP/PSP 


Post-Release  Defects/KLOC  -  Range 


o_  iv  a  i  i /c  ^  i  o  non  TP  nit; 
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TSP  Performance  Comparison 


Performance 

Category 

Typical 

Industry 

Performance1 

Study 
Baseline 
(2000)  2 

TSP  Impact 
Study  (2000)3 

TSP  Impact 
Study 
(2003)4 

Schedule  Error 

180%  avg. 

27%  to  112% 

5%  avg. 

6%  avg. 

Effort  Error 

130%  avg. 

17%  to  85% 

-4%  avg. 

26%  avg. 

System  Test  defects 
per  KLOC 

5  to  25  approx. 

NA 

0.0  to  0.9 

0.4  avg. 

0.0  to  0.9 

Released  defects  per 
KLOC 

1  to  7 

0.2  to  1  + 

0.0  to  0.35 

0.06  avg. 

0.0  to  0.2 

1.  Gartner  Group 

2.  Control  projects  from  CMU/SEI-2000-TR-015,  a  study  of  18  TSP  projects  in  four  organizations  conducted  in  2000. 

3.  TSP  projects  from  CMU/SEI-2000-TR-015,  a  study  of  18  TSP  projects  in  four  organizations  conducted  in  2000. 

4.  TSP  projects  from  CMU/SEI-2003-TR-014,  a  study  of  20  projects  in  13  organizations  conducted  in  2003. 
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TSP  Improves  Quality 


An  analysis  of  20  projects  in 
13  organizations  showed 
TSP  teams  averaged  0.06 
defects  per  thousand  lines 
of  new  or  modified  code. 


Approximately  1/3  of  these 
projects  were  defect-free. 


These  results  are 
substantially  better  than 
those  achieved  in  high 
maturity  organizations. 
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TSP  Principles 


Software  is  developed  by  individuals  and  teams.  To  improve  overall 
performance,  improve  individual  and  team  performance. 

Empowered  teams,  coaching,  and  rational  management  are  the  most 
effective  way  to  manage  engineering  teams. 

Process  is  a  performance  determinant... an  ad-hoc  chaotic  process  cannot 
produce  predictable,  high-quality  results. 

Measurement  is  necessary  for  high-performance. 

To  get  a  quality  product  out  of  test,  you  must  put  a  quality  product  into  test. 
Quality  must  be  measured  and  managed  at  every  step. 
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Building  High-Performance  Teams 


The  TSP  strategy  for 
improving  performance  is  to 
build  high-performance  teams 
from  the  bottom-up. 


Team 

Software 

Process 


Team 

Management 


Team  communication 
Team  coordination 
Project  tracking 
Risk  analysis 


Team 

Building 


Personal 

Software 

Process 


Team 

Member 

Skills 


Goal  setting 
Role  assignment 
Tailored  team  process 
Detailed  balanced  plans 

Process  discipline 
Performance  measures 
Estimating  &  planning  skills 
Quality  management  skills 
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The  TSP  Launch  Process 


Day  1  Day  2  Day  3  Day  4 
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TSP  Process  Structure 


The  TSP  process  elements  can  be 
organized  into  whatever  process 
structure  makes  the  most  business 
and  technical  sense. 


The  phases  can  be  implemented 
iteratively  in  small  cycles,  in  a  spiral 
with  increasing  cycle  content,  or 
sequentially  as  in  a  waterfall, 


TSP  projects  can  start  on  any 
phase  or  any  cycle. 


Each  cycle  starts  with  a  launch  or 
re-launch  and  ends  with  a 
postmortem. 
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Estimates,  plans, 
process,  commitment 


Lessons,  new 
goals,  new 
requirements, 
new  risk,  etc. 


Development 
phase 
or  cycle 


Phase  or  cycle 
Postmortem 


Work  products, 
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Reasons  for  extending  the  TSP  Method  to 
the  Systems  Engineering  Domain 


Cost-benefit  numbers  exist  from  TSP  on 
software  projects 

Requests  from  existing  product  teams  using 
TSP  to  extend  to  their  Systems  Engineering 
(and  other)  process  areas 

Increasingly  difficult  to  solve  software  problems 
without  addressing  systems  engineering  issues 
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Extending  TSP  Method  to  the  SE  Domain — 
Team  Process  Integration  (TPI ) 


How  do  we  tailor  TSP  for  other,  non-software  disciplines? 
The  research  challenges  include: 

•  Processes 

•  Measurement 

•  Role  Definition 

•  Training 

•  Tool  Support 
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Research  Challenges  -  Processes 


-  Develop  prototype 
processes/scripts 
for  SE  and  other 
engineering 
disciplines 


-  Use  “traditional” 

TSP  Launch  Process 
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Research  Challenges  -  Measurement  - 1 

Schedule  and  effort  measures  are  essentially  unchanged. 

Lines  of  Code/Function  Points  would  not  serve  as  relevant  size 
measures  for  SE.  Formulate  size  measures  for  SE.  Examples: 

•  DOORS  objects 

•  Requirements 

•  Items  Verified 


Size 


Effort 


Schedule 


Quality 
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Research  Challenges  -  Measurement  -  2 


Quality  measures  in  SE 

•  Define  what  “quality”  means  in  SE 

•  Where  in  the  process  do  you  collect  data? 

•  What  are  the  derived  quality  measures  (e.g., 
defects/DOORS  object?) 

•  Establish  initial  quality  baseline 
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Research  Challenges  -  Measurement  -  3 


What  are  the  quality  goals? 

Examples: 

•  Goal:  Accuracy  in  the  work 

Measure:  #  of  problem  reports  against  requirements 
and  test  documents 

•  Goal:  Conformance  to  standards 

Measure:  #  of  defects  in  peer  reviews;  #  of  defects 
in  requirements  and  test  documents,  etc... 
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Research  Challenges  -  Role  Definition 


Apply  four  primary  roles — planning,  process,  quality, 
support 

Assess  applicability  of  remaining  roles  and  define 
additional  roles  needed  for  SE. 

•  Add  Requirements  Manager 

•  Combine  Design  and  Implementation  roles  into 
one  role 

•  Expand  Test  Manager  role  as  needed,  e.g., 

—  Flight  Test  Manager 
—  Lab  Test  Manager 
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Research  Challenges  -  Training  - 1 


Currently,  training  is  geared  to  software  teams. 

So  challenges  include: 

•  building  conviction  and  discipline  in  teams  that 
don’t  write  software  programs 

•  providing  just  the  right  amount  of  training  to  get 
a  team  started 

•  supplementing  with  additional  training  modules 
as  needed 
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Research  Challenges  -  Training  -  2 

Develop  “JIT”  training  to  support  SE  teams 

Develop  Leadership  Seminar  and  Team  Member  Training  to 
focus  on: 

•  providing  the  fundamentals  of  TPI 

•  launching  a  team 

•  maintaining  a  plan 


Follow-up  with  additional,  “JIT”  training,  e.g., 

•  Inspections 

•  Measurement,  data  analysis,  and  reporting 

•  Checkpoints  and  Postmortem  Analysis 

•  Tool 


Research  Challenges  -  Support  Tool 


Develop  an  extensible  tool  that  allows  for: 

•  Defining  any  process 

•  Collecting  data  unobtrusively 

•  Defining  a  measurement  framework 
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TPI  Project  Goal 


Extend  TSP  method  and  practice  to  other  domains 

•  Demonstrate  that  distributed  teams  using 

Tailorable  process  scripts 
—  Metrics 

Automated  tool  support 

can  deliver  quality  products  and  services  on  cost 
and  schedule 
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TPI  Fundamentals 


To  be  successful,  teams  need 

•  clear  goals  and  roles 

•  agreed-upon  processes  and  plans 

Build  self-directed  teams 

•  team  members  own  their  own  processes  and  plans 

•  team  members  understand  their  goals  and  roles 

•  team  members  voluntarily  commit  to  their  goals 

•  team  members  believe  their  commitments  are  achievable 

The  TPI  launch  builds  self-directed  teams. 

Team  members  of  self-directed  teams  practice  disciplined 
personal  processes. 

TPI  data  rolls  up  to  the  project  and  program  level. 
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TPI  Systems  Engineering  Pilots 


Conduct  a  series  of  pilot  projects  to  determine  if  extending 
TSP  practices  to  Systems  Engineering  and  other  engineering 
disciplines  results  in  measurable  improvement 

•  Engineering  Discipline  A 

•  Engineering  Discipline  B 

•  Systems  Engineering/Integration 

•  Product  Integrity 

•  Lab  Team 

•  Test  Team 

Use  these  results  to  establish  common 
processes/measures  within  and  across 
projects/programs 
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Pilot  TPI  Implementation  Approach  at  Org  A 
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Preliminary  Results  from  Org  A 


Cumulative  Planned  and  Actual  Hours  per  Week 


12000.0 


10000.0 


-  Cumulative  Planned  Hours 
Cumulative  Actual  Hours 
Baseline  Cumulative  Plan  Hours 


loHo 

Weeks 


Team  has  been  consistently  meeting  or  exceeding  planned  task 
hours  (Faltering  EV  trend  is  not  due  to  failing  to  meet  task  hour 
commitment.) 
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Org  A  Qualitative  Benefits  - 1 


High  value  in  launch  process  -  management  communicating 
plans  and  priorities,  team  developing  plans,  coordinating  & 
integrating  across  program/project  teams 

Tracking  the  work  is  helping  team  members  to  stay 
on  track 

•  “I  need  to  stop  doing  X  to  get  back  on  track.” 

•  Very  easy  to  see  the  impact  daily  and  weekly  of  not 
working  to  the  plan 

•  Realization  that  one  can’t  keep  doing  extra  tasks 
without  impact  to  the  plan 
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Org  A  Qualitative  Benefits  -  2 


Team  meeting — weekly  coordination/communication  with  team 
members  and  off-loading/balancing  of  tasks 

Team  members  functioning  as  a  self-directed  team  by  taking 
on  team  management  roles--Planning  Manager,  Process 
Manager,  Test  Manager,  and  Lab  Manager  roles  being 
executed 

Better  estimates  for  second  launch  cycle  based  on  having  . .  . 
•Common  processes 
•Common  language 

•Common  metrics  . .  .  within  team  and  across  teams  and 
the  entire  program 
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Lessons  Learned 


First  Launch  -  estimates  based  on  time,  experience, 
and  engineering  judgment 

Second  Launch  -  estimates  based  on  data  from  the 
first  launch  and  proxies  developed  for  systems 
engineering  work 

•  Resulted  in  plan  conning  in  much  closer  to  desired  schedule 
dates 

•  Better  able  to  define  work 

•  Agile  team  able  to  adapt  easily  to  program/personnel 
changes — i.e.,  saves  the  team  six  months  of  defining  new 
work  for  new  team  member 

•  Making  course  corrections  as  needed 
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Team  Leader  Feedback  from  Org  A 


“Prior  to  TPI,  we  made  estimates  in  a  bubble.” 

“Now  we  are  establishing  and  maintaining  baselines 
for  all  of  our  releases  which  allows  us  to  make  better 
estimates  and  more  realistic  plans  and  schedules.” 

“We  have  a  team  concept  across  our  program  with  all  of  the  subteams 
(SE,  PI,  SW,  Test,  Lab,  ...).  We  also  have  a  common  set  of  processes 
and  metrics  to  help  all  of  the  teams  better  communicate  and  address 
dependencies  across  the  teams.” 

“We  are  starting  to  roll-up  our  team  data. 

Next,  we  need  to  roll-up  data  across  all  of  the  teams  in  the  XYZ  Program.” 


How  To  Talk  To  A  Program  Manager 


Software  Engineering  Institute  CarnegieMellon 


©2007  Carnegie  Mellon  University 


77 


Pilot  Approach  at  Org  B 


Overall 
Project 
SE  Product 


SE  Team  1 

SE  Team  2 

SE  Team  3 

SE  Team  4 

Component 

Component 

Component 

Component 

•Using  TPI  for  2+  yrs 

•Extended  to  6  teams  and  growing 
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Preliminary  Results  from  Org  B  (SE  Domain) 


Cumulative  Earned  Value 
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Another  example  from  Org  B 


Cumulative  Planned  and  Actual  Hours  per  Week 
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Comments  on  Previous  EV  Chart 


Schedule  performance  diverged  from  the  plan  at  the  time  this 
sample  was  taken. 

The  team  was  unable  to  apply  the  planned  hours. 

This  often  results  from  resource  allocation  to  other  projects. 

This  information  permits  the  project  manager  and  the  program 
manager  to  make  informed  decisions  when  allocating  scarce 
resources  among  multiple  projects. 
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TPI  Approach  Shows  Promise 


Eights  teams  are  pilot  testing  TPI 

The  launch  process  is  applicable  in  any  engineering  domain  to: 

•  produce  an  executable  team  plan 

•  get  management’s  agreement  to  the  plan 

•  to  build  an  enthusiastic,  self-directed  team 

Provides  data  that  PM’s,  team  leads,  team  members  can  use 
Provides  common  language  and  framework  across  teams 
A  support  tool  is  being  prototyped 

•  based  on  MS  Access 

•  can  be  adapted  to  any  engineering  discipline 

Training  has  been  developed  and  tailored  for  SE  and  other  teams 
(e.g.,  Senior  Leadership  Training,  Team  Member  Training,  Tool  Training) 
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What  makes  TPI  unique — 
Putting  the  pieces  together 


You  may  already  be  using  some  of  the  planning,  tracking,  and  quality  management 
practices  in  the  TPI. 

What  makes  the  TPI  unique  is  that  all  these  practices  are  integrated  into  a  simple, 


And  that  framework  can  not  only  be  used  at  the  project  level,  but  at  an  individual 
team  member  level  to  guide  day-to-day  work. 
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Improving  How  YOU  Talk  to  a  Program  Manager 


Reliable  size,  effort,  cost,  schedule,  and  quality  data  are  required. 

You  get  what  you  ask  for,  so  know  what  to  ask  for  and  how  to  use  what 
you  get. 


Remember — if  you  try  you  might  just  get  what  you  need 

Detailed  plans  that  support  weekly  tracking  and  reporting  of  these  data  are 
necessary  to  detect  trouble  early. 


Know  how  to  use  the  data  to  identify  the  real  issue,  define  a  realistic 
course  of  action,  and  take  it. 


Team  Process  Integration  shows  promise  as  means  to 
improve  How  YOU  Talk  to  a  Program  Manager! 
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For  Further  Information  . . . 

If  You  Are  Interesting  in  Piloting  TPI 


John  Mishler  imishler@sei.cmu.edu 
Anita  Carleton  adc@sei.cmu.edu 
Bill  Nichols  wrn@sei.cmu.edu 

Carnegie  Mellon  University 
Software  Engineering  Institute 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 
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Event  Time  Analysis  in  Multi  Mission 
Scenarios  with  System  Simulation  Models 


Ravi  Moorthy 
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Topics 


•  Abstract 

•  Definitions 

•  Multi  Mission  Scenarios  and  Missions 

•  Analysis  Methodology 

•  Event  Timeline 

•  Simulation  Models 

•  Performance  Metrics 

•  Sensor  Resource  Analysis 

•  System  Concept  Development  Derivation 


Abstract 


This  paper  presents  an  event  time  line  analysis 
using  system  simulation  models  in  multi  mission 
scenarios 

Radar  resource  usage  is  evaluated  for  the 
assessment  of  the  multi  mission  capability  of  the 
comba  t  system 

The  impact  of  radar  resource  availability  is 
evaluated  in  scheduling  the  events  along  the 
engagement  timeline  using  the  system  simulation 
models 
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Definitions 


•  AAW  -  Anti  Air  Warfare 

•  Sensors  -  Scanning  and  Tracking  Radars 

•  Multi  Mission  -  Combined  Missions  requiring 
Simultaneous  Tracking  and  engagements  for 
Ballistic ,  Air  or  Surface  Targets 

•  Radar  Resource  Usage  -  Radar  time  required  to 
schedule  the  beams  for  search  and  track 
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Scenarios  and  Mission 


General  Engagement  Timeline  Model 


Report 
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Multi-Mission  Scenario 


•  Goal 

-  Assess  Naval  Ship  Combat  System 
capability  to  simultaneously  perform 
multiple  Missions 

•  Typical  Operational  Scenario 

-  A  single  ship  is  tasked  with  both  mission  A 
and  a  mission  B 

-  The  ship  will  require  management  of  radar 
resources  to  achieve  both  missions 
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Analysis  Methodology 


•  Mission  specific  system  simulation  models 
are  used  individually  to  determine  individual 
mission  performance 

•  With  post  processing  the  radar  resource 
usage  is  analyzed  for  combined  event  time 
lines  of  the  missions 

•  Alternate  system  concepts  can  be  derived 
from  the  analysis 

•  By  managing  priorities  with  some  of  the 
events,  radar  resources  can  be  managed  to 
provide  capability  for  both  missions 
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Multi-Mission  Capability 
Analysis  Methodology 


Raid  of  missiles  arriving  at  a 
given  interval  inbound  and  in  the 
vicinity  of  the  ship  while  search  is 
performed  at  a  given  rate 


Raid  of  missiles  arriving  at  a 
given  interval  from  a  given 
launch  area  impacting  at  a  given 
distance  from  ownship 


Mission  A 


Mission  B 


%  Radar  Usage 
Required  for 
Mission  A 
Capability 


%  Radar  Usage 
Required  for 
Mission  B  Capability 


Combined  analysis  results  for  multi-mission  scenario 
along  the  event  timeline  to  evaluate  whether  a  single  ship 
system  can  simultaneously  perform  A  and  B  missions 
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Simulation  Models 


•  System  simulation  models  specific  to  missions 
were  used 

-  Event  driven  simulation  models 

-  High  fidelity  representation  of  radar  and 
weapon  systems  in  the  models 

•  System  simulation  models  were  run  individually 
for  mission  A  and  mission  B 

•  Output  data  was  combined  for  event  time  analysis 

•  Post  processing  of  the  data  with  MATLAB  and 
Excel  scripts 
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Resource  Analysis  Methodology 


•  Event  timeline  versus  available  sensor  resources 

•  Intersection  of  multiple  events  create  stresses  on 
sensor  resources 

•  Analyze  the  initial  intersection  of  events  against 
mission  priority  to  determine  unsupportable  events 

-  Event  set  1  is  higher  priority  then  event  set  2 

-  Event  set  1  is  given  the  system  resources 

•  Determine  baseline  measurement  of  effectiveness 
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Resource  Analysis  Methodology 


•  Determine  if  sufficient  time  is  available  to 
reschedule  an  unsupportable  event. 

-  If  not  reschedule-able,  then  event  timeline 
is  not  supportable 

•  Shift  event  data  (apply  system  concept)  and 
re-analyze  remaining  events 

•  Determine  applied  system  concept 
measurement  of  effectiveness 
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Radar  Resource  Usage  in  a  Mission  B 
Scenario 


Missile  in  Flight 


Sensor 

Resource 

Tracking 

1 

'◄ - 

Tracking  &  Mis 
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Performance  Metrics 


•  Radar  resource  usage  along  engagement 
event  timeline 

•  Number  of  supportable  engagement  events 
along  the  timeline 

•  Number  of  targets  not  engaged  due  to 
resource  limitations 


Filename,  5/3/01  14 


Derived  System  Concept 


•  The  engagement  event  time  lines  for  the  mission 
are  built  from  the  output  data  of  the  Monte  Carlo 
Simulation  runs 

-  Determine  peak  supportable  events  and 
system  resource  usage 

-  Determine  total  supportable  event  timelines 
allowing  for  rescheduling 

•  Derive  observations  on  system  concept 
functionality  required  to  support  a  revised  total 
supportable  event  timeline 
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Overview 


■  Definitions 

■  Leadership/Management/Technical 
(dis)  Harmony 

■  Precedented/ Unprecedented  systems 

■  Examples 

■  Methods 

■  Leadership  evaluation 

■  Mapping  leadership  styles  to  precedented  and 
unprecedented  systems 

■  Mitigating  style  mismatches  for  individuals 

■  Examples  of  mappings  and  mitigation 
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Definition:  Leadership/ Management/ 
Technical  (Dis)  Harmony 


M.  Robertson 
Head  of  MP3.com 


Leadership  Style 


JteVe  Jobs 
/Applt 


Sen  Billy  Mitchel 
rather  of  US  airpower 


Col  R.  Smith 

Leader  of  Bradley  Program 
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(Defense  Acq  Univ) 

Heirafrchical  verses  flat 

Management 

Structure  Environment 
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DoDAF  \  Bradley  AFV 

Technical 
Execution 


Definition:  Leadership 


■  Transactional  occurs  when 

■  “Leader  rewards  or  disciplines  the  follower  depending  on  the  adequacy 
of  the  follower's  performance 

■  Contingent  reinforcement,  either  the  positive  of  contingent- reward  or 
more  negative  of  active  or  passive  forms  of  management  by 
exception" 

■  Transformational  is  seen  when 


■  'They  motivate  others  to  do  more  than  they  originally  intended  and 
often  more  than  they  thought  possible 

■  Stimulate  interest  among  colleagues  and  followers  to  view  their  work 
from  new  perspectives 

■  Generate  awareness  of  the  mission  or  vision  of  the  team  or 
organization 

■  Develop  colleagues  and  followers  to  higher  level  of  ability  and  potential 


■  Motivate  colleagues  and  followers  to  look  beyond  their  own  interests 
toward  those  that  will  benefit  the  group" 

-  Bass,  B.  M.,  Avolio,  B.  J.,  Jung,  D.  I.,  &  Berson,  Y.  (2003).  Predicting  unit  performance  by  assessing  transformational  and 
transactional  leadership.  Journal  of  Applied  Psychology,  88(2),  207-218.  (Reprinted  with  the  permission  of  Mind  Garden,  \r^.) 
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Definition:  Stresses  the  Move 
to  Disharmony 


■  Budget  mandates 

■  New  technology  maturations 

■  I  ndependent  reviews 

■  Performance  of  peer  leaders/ managers 

■  New  requirements  and  requirement 
combinations  that  make  a  system 
unprecedented 
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Definition:  Precedented  and 
Unprecedented  Systems 


■  Precedented  System 

■  Technically  similar  to  one  that  has  been  built  before 

■  Schedule  and  budget  constraints  (and  hence 
productivity)  are  similar  to  previous  efforts 

■  Unprecedented  System 

■  Technically  dissimilar  to  previous  systems 

■  Even  if  technically  similar,  faster  schedules  or  lower 
budgets  than  previous  efforts  may  require  sufficient 
innovation  in  the  delivery/ productivity  to  make  the 
development  of  the  system  unprecedented 

-  Sisti,  F.  J.  &  Latimer,  D.  T.  (2007).  Linking  leadership  and  technical  execution  in  unprecedented 
systems-of-systems  acquisitions.  Journal  of  Integrated  Design  and  Process  Science,  (In  Press). 
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Example:  Precedented System 
with  Transactional  Leadership 


■  Demonstrates  traction  and  harmony  from 
leaders  and  management  since  they  know 
what  to  direct  the  technical  folks  to  do 
and  have  reasonable  expectations  of 
success  of  assigned  tasks;  conversely, 
technical  folks  know  what 
management/ leadership  wants  (the 
precedented  system  using  precedented 
methods) 


SEC  10/25/2007 
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Example  -  Armor  kits  for  the  HMMVs  in 
I  raq  and  Afghanistan 
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Example:  Unprecedented  System 
with  Transactional  Leadership 


■  MP3.COM 

■  Michael  Robertson,  CEO 

■  MP3  Technology/ 1  nternet  Based 

■  Artist  and  Customer  friendly 

■  2000,  Universal  Music  Group  -  $250  (T) 

■  Forced  Sales 

■  Demonstrates  stresses  that  couldn't  be 
handled  transactional ly that  eventually  led 
to  loss  of  the  company 


VISIO 
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Example:  Unprecedented  System 
with  Transformational  Leadership 


■  PMA-271  -  E-6B  TACAMO  Program  -  combination 
ABNCP  and  Longwire  VLF  antenna  onto  single 
Boeing  707-320 

■  Two  systems  never  before  combined 

■  Combined  services  (Navy  and  USAF  programs)  to 
consolidate  C&C  of  strategic  assets  across  two  services 

■  Various  challenges  -  Technical  &  Management 

■  PM  demonstrated  the  41  's  of  Transformational  Leadership: 

■  I  dealize  I  nfluence, 

■  Inspirational  Motivation, 

■  Intellectual  Stimulation,  and 
I  ndividualized  Consideration 


VISIO 
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A  Way  Ahead? 


Once  you  understand  the  system 
impact,  how  can  you  modify  your 
leadership  to  improve  chances  of 
success? 


SEC  10/25/2007 
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Methods:  Leadership  Evaluation 
Mechanisms 

■  Multi-Factor  Leadership  Questionnaire®  (MLQ®) 

■  Heritage:  research  in  1978  by  Burns  extended  in  1985  Bass 

■  What  is  measured:  7  component  model  of  leadership 

■  How  it  is  measured:  45  questions,  360-degree  preferred, 
reported  out  across  the  7  leadership  behaviors 

■  Full  Range  of  Leadership®  (FRL®) 

■  Heritage:  1999  Bass/Avolio  extended  MLQ®  to  give  leaders 
direction  to  improve  their  preferences 

■  What  is  measured:  7  leadership  behaviors 

■  How  is  measure  used:  Augmented  MLQ®  report  provides 
input  to  increase  specific  leadership  capabilities 


-Avolio,  B.  J.  &  Bass,  B.  M.  (2006).  Multifactor  leadership  questionnaire  (3rd.).  Menlo  Park,  CA:  Mind  Garden,  Inc. 
(Reprinted  with  the  permission  of  Mind  Garden,  Inc.) 
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The  Model  of  the  Full  Range  of 
Leadership® 


The  four  fs: 

Idealized  Influence  (Charisma) 
Inspirational  Motivation 
Intellectual  Stimulation 
Individualized  Consideration 
Contingent  Reward 
Management-by-Exception  -  Active 
Management-by-Exception  -  Passive 
Laissez-Faire  Leadership 


EFFECTIVE 


MBE-A 


CR 


Is 


ACTIVE 


O 


•H 


(INEFFECTIVE 

Bass,  B.  M.,  &  Riggio,  R.  E.  (2006).  Transformational  leadership:  Second  edition  (Rev.  e<±).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 
(Reprinted  with  the  permission  of  Mind  Garden,  Inc.)  12 
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Comparing  the  Two  Points-of-View 


Strength 

Comparisons 


Transactional 

■  Maintains  subordinate  levels  & 
grows  individual  experience 

■  Focuses  on  "wait  for  direction" 
work  ethic 

■  Encourages  linear  actions 
focusing  on  extending  planned 
schedules 

■  Fosters  point-to-point  solutions 

■  Limits  perception  of  value  to 
overall  mission  success  and 
effectiveness 

■  Provides  individual  with  narrow 
experience  profile 

■  Does  not  encourage  trust 

■  Does  not  require  much  training 
to  maintain  competency 


System 

Challenges 


Precedented 

■  Technically  similar  to  one  that 
has  been  built  before 

■  Schedule  and  budget  constraints 
(and  hence  productivity)  are 
similar  to  previous  efforts 


SEC  10/25/2007 
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T ransformational 

■  Builds  subordinate  capabilities  & 
potential  through  experiences 

■  Builds  understanding,  morale,  & 
trust 

■  Encourages  multi-linear 
capability  focusing  on 
maintaining  or  reducing 
schedules  ' 

■  Fundamentally  net-centric 
aware 

■  Enables  perception  of  value  to 
overall  mission  success  and 
effectiveness 

■  Provides  capacity  for  transfer  of 
knowledge 

■  Requires  trust 

■  Requires  appropriate  training 

Unprecedented 

■  Technically  dissimilar  to 
previous  systems 

■  Even  if  technically  similar,  faster 

schedules  or  lower  budgets  than 
previous  efforts  may  require 
sufficient  innovation  in  the 
delivery/  productivity  to  make 
the  development  of  the  system 
unprecedented  i 3 


I 


Method:  Individual  Development 
Plans  for  Leaders/ Managers 


■  Setting  goals 

■  Approach  should  be  based  on  system  type 
and  FLR*  results,  style,  current  environment, 
personal  preferences,  and  career  planning 

■  With  360  feedback,  you  know  if  you  are 
being  successful  with  your  leadership  efforts 

■  What  you  desire  in  the  future  environment 
becomes  the  goal 


-Avolio,  B.  J.  &  Bass,  B.  M.  (2006).  Multifactor  leadership  questionnaire  (3rd.).  Menlo  Park,  CA:  Mind  Garden,  Inc. 
(Reprinted  with  the  permission  of  Mind  Garden,  Inc.) 
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Poi  nts- to-  Ponder 


■  This  presentation  addresses  a 
current  examination  of  the  interplay 
between  leadership  and 
effectiveness/  success 

■  Conclusion  is  that  failure  to  adapt 
leadership  to  the  circumstances  of 
your  project  or  organization  can  lead 
to  unexecutable  programs 


VISIO 
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Additional  Sources  -  1 


■  Avolio,  B.  J .  &  Bass,  B.  M.  (2006).  Multifactor  leadership 
questionnaire (3rd.) .  Menlo  Park,  CA:  Mind  Garden,  Inc. 

■  Avolio,  B.  J .,  Waldman,  D.  A.,  &  Yammarino,  F.  J .  (1991). 
Leading  in  the  1990s:  The  four  I 's  of  transformational 
leadership.  Journal  of  European  Industrial  Training,  13,4) ,  9-16. 

Bass,  B.M.  (1999).  Two  decades  of  research  and  development  in 
transformational  leadership.  European  J  ournai  of  Work  and 
Organizational  Psychology,  8,  9-32. 

Bass,  B.  M.  (1998).  Transformational  leadership:  Industrial, 
military,  and  educational  impact.  Mahwah,  NJ :  Lawrence 
Erlbaum  Associates. 

Bass,  B.M.  (1990).  Bass  &  Stogdill's  handbook  of  leadership: 
Theory,  research  &  managerial  applications.  (3rd  Ed.).  New 
York:  The  Free  Press. 


Bass,  B.  M.  (1985).  Leadership  and  performance  beyond 
expectation.  New  York:  Free  Press. 


Bass,  B.  M.,  Avolio,  B.  J .,  J  ung,  D.  I .,  &  Berson,  Y.  (2003). 
Predicting  unit  performance  by  assessing  transformational  and 
transactional  leadership .  J  ournai  of  Applied  Psychology,  88,1), 
207-218. 
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Additional  Sources  -  2 


Bass,  B.  M.,  &  Riggio,  R.  E.  (2006).  Transformational  leadership: 
Second  edition  (Rev.  ed.).  Mahwah,  NJ :  Lawrence  Erlbaum 
Associates. 

Bums,  J .  M.  (2003).  Transforming  leadership.  New  York:  Grove 
Press. 

Burns,  J .  M.  (1978).  Leadership.  New  York:  Harper  &  Row. 

Mintzberg,  H.,  Lampel,  J .,  Quinn,  J .  B.,  &  Ghoshal,  S.  (2003).  The 
strategy  process  -  concepts,  contexts,  cases.  Upper  Saddle  River, 
NJ :  Prentice  Hall. 

Eid,  J .,  J  ohnsen,  B.  H.,  Brun,  W.,  &  Laberg,  J .  C.  (2004).  Situation 
awareness  and  transformational  leadership  in  senior  military 
leaders:  An  exploratory  study.  Military  Psychology,  16(3),  203-209. 

■  Pinney,  C.  W.  (1999,  Winter).  The  USAF  PEO/DAC/MAD  structure: 
Successful  pattern  for  future  weapon  system  acquisition? 
Acquisition  Review  Quarterly  21-  46. 

Sisti,  F.  J .  &  Latimer,  D.  T.  (2007).  Linking  leadership  and  technical 
execution  in  unprecedented  systems- of- systems  acquisitions. 
Journal  of  i nteg rated  Design  and  Process  Science,  ( I  n  Press) . 
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<\sset-Based  PBL  for  Navy  Warships  ~  A 
case  study  for  LCS  Class  Shi  pi 


O 


NDIA  -  10cn  Annual  Systems  Engineering 

Conference 

Oct  24,  2007 


Mike  Mahon 


j  Definitions/PBL  scaling 
j  LCS  Overview 


J 


J 


J 


Asset-Based  PPL 
Asset-Based  PPL 
Asset-Based  PEL 
Path  ahead 


Key  questions 
challenges/ Obsta  cles 
-  keys  to  success 


J 


Definitions 


What  is  PEL? 

°  any  contract  where  the  primary  requirement  is  to 
provide  products  &  services  bused  on  a  pre¬ 
determined  performance  metric. 

•  The  performance  metric  should  in  some  way  be  a 
con  tributing  factor  to  Operational  Availability  (Ao). 


The  Navy  today  boosts  of  150 +  PEL  con  trusts;  most 
of  these  ore  supply-oriented  PELS  issued  by 
NA  VICE 

•  Most  ore  lower  level  component  bused  PELs 


Definitions  (coni) 


What  is  Asset-b  sed? 


Component  Level 


Easier 


Complexity  is  based  of  number  of 
systems  within  asset 


- ► 

Harder 


PBL  Complexity 


There  sure  two  primary  dimensions  to  PBL  complexity. 


PBL  Complexity  Scaling 


Note:  this  scale  is  Mahon  developed  and  not  an  industry  accepted/certified  rating  system  for  PBLs 


LCS  consists  of  core  seaframes  design  ad  to  /lost 
mission  packages.  Three  MP  are  initially  planned. 


MIW 


VTUAV  (2) 

H 

COBRA  (2) 

RMS  (2) 


— 5  OASIS 


ASW 


ASUW 


MH60R  (1) 


onobuoyM«54 
Torpedo 


ALFS 


Multi-Static  Lightweight  Array 
Active  Source 


RTAS 


MFTA 


EO/IR  GAU  21  Hellfire 
Gun 

NLOS-LS  (4) 


30mm  Gun  (2) 


Requirements 

THRESHOLD 

OBJECTIVE 

Sprint  Speed  (kts) 

40 

50 

Mission  Package  Payload 
(mt) 

180 

210 

Range  @  Transit  Speed 
(nm) 

3500 

4300 

Navigational  Draft  (ft) 

20 

10 

Core  Crew  manning 

50 

15 

Twd  Yslffs  S220M 

Any  Mission  Package  /  Any  Ship  /  Any  Time 


•  Non-traditional  hull  forms 

•  Non-traditional  materials 

•  Non-traditional  Propulsion 

-  CODAG  +  Waterjet  drive  (x4) 

•  Non-traditional  construction  practices 

•  Non-traditional  system  suppliers 

•  Modular  Open  Systems  Approach 

•  Open  Computing  Architecture 

•  Automation 


LCS  Breaks  thru  many  Traditional  Paradigms 


LCo 


;  Flight  0  Acquisition 


6  Industry  Concepts 

(06  Feb  03) 


February 

2003 


3  Preliminary  Designs 

(19  Jul  03) 


July 


2  Final  Designs 

(Contracts  Awarded  27  May  04) 


} 

_ 


FLT  0  Lockheed 
Martin 
(15  Dec  04) 


May  December 


2003 


2004 


2004 


FLT  0  General 
Dynamics 

(14  Oct  05) 


October 

2005 


First  Ships  Delivers  in  Summer  2008 


LCS  Flight  0  Sustain  merit  Strategy 


The  US  Navy  approach  for  LCS  sustainment  is  to 
establish  the  held  shipbuilding  hums  as  hud  for 
sustainment  for  mi  interim  36-month  period. 

—  Concept  is  to  leverage  know! edge  for  design/ construct! on  for  risk 
mitigation  in  initial  sustainment  phase 


2009 


2010 


2011 


Lockheed  Martin 
Team 


General  Dynamics 
Team 


1.  Will  it  work?  Just  because  it  works  at  lower  level-: 
doesn't  mean  it’s  a  good  thing  at  a  higher  level? 


2,  Will  it  save  m  on  ey  an  d  if  soj  how  mu  oh  ? 


J. 


Cun  we  really  put  such  heavy  responsibility  for 
our  nations  defense  in  the  hands  of  Industry? 


4,  What  is  the  fallback  If  it  doesn't  work? 


5.  What  will  happen  to  the  existing  Infrastructure  that 
Is  still  repaired  for  other  ship  classes? 


Asset-based  PBL  -  Oharenges/Obstaclei 


1 

2 


Ld  j 


A 

SJ 


o. 


o. 


Jobs/responsibilities  —  sue  ettuchmd 

Risk —  nil  dimensions  of  risk  mus  t  be  identified  end 

mitigation  plans  established  and  funded 

Colors-of-money  (RDT&E,  SCN,  O&M,  mud) 

•  /f  /s  difficult  securing  on  extra  SCN  dollar  to  save  two  d oilers  of 
OM&N 

Cost/ business  esse  analyses  (b  lj\ J 

•  Most  transformational  concepts  require  e  BCA,  yet  establishing 
e  baseline  for  today’s  worships  is  difficult  at  best 

Interaction  with  other  existing  PbLs 

•  deed  to  ensure  that  upper  level ,  asset-based  PbLs  con  work  In 
harmony  with  existing ,  established  PbLs 

Patience  (or  hick  of  it) 

°  Initial  performance  will  be  bumpy/full  of  glitches  -  ell  peril es 
need  to  be  prepered  for  this  end  work  through  it 


Asset-based  PBL  -  Keys  to  Success 


1 

2 


'j 


// 

-y« 


Support  from  DoD/Customer  community 
A  Good  approach  that  manages  Risk 

•  ‘stair-step9  approach  that  progresses  to  full  asset-based 
PBL  incrementally 

°  Initially  costs  mors  to  have  parallel  paths  In  csss  of 
failure 

°  Integrated  industry- Go  vt  proa 

So/icf  team  Structure 


•  Embraces/uses  competition  fur  optimal  value 

Good  performance  metrics 


Path  Ahead 


1.  Build  the  team  and  die  processes  for  the  three 
your  Interim  Sustainment  timeframe. 

2 Establish  initial  metric  set 

3,  Do  NOT  accept  PBL  from  Initial  suppliers  - 
risk/cost  will  be  too  high.  Instead  use  the  3yr 
period  to  understand  the  ship  and  operational 
caps  and  lims  “  measure  everything 1 

4.  Build  alternate  suppliers  —  keep  competitive 


v. 


en  vironment 

Establish  transition  plan  for  full  life-time  support 
(Also  build  phm  to  fallback  to  traditional  approach  If  reqd) 


Back-up 


Job/Responsibilitm 


The  biggest  o bstacie  to  asset-b  used  pel  [or  Esc 
or  CL  S)  will  be  from  the  organic  support 
infrastructure  who's  very  livelihood  is  threatened 
by  this  initiative 

Unlike  component  based  PEL  f which  never  shifted 
'who  did  the  work  but  how  it  was  contracted), 
asset-based  PBL  transitions  organic  responsibility 
to  industry 

And  yet,  industry  must  work  with  those  very  sum e 
organic  activities  to  develop  end  operate  the 
Asset-based  PEL 

Many  people/organizations  will  be  very  happy  to 
see  asset-based  PBL  full  end  may  even  work  to 
help  it  hill 


Asset-Based  PBL  -  Org  structure 

•  The  business  structure  of  an  asset-based  PBL  for  a  warship  cun  be 
very  complex .  It  consists  of  many  suppliers  end  varying  levels 
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Performance  Metrics 


j  Measuring  performance 
is  critical 

SCM 

Maintenance 

Training 

■Inventory  management 

•Casualty  response  time 

•Train-to-qualify  (T2Q) 

•  Samples  mom  os 

■Demand  forecasting 

•Remote  monitoring 

•Embedded  training 

include: 

•Transportation 

•Condition-based  Maint. 

*lnitial&  replenishment 

■Requisition  processing 

•Distance  support 

crew  training 

•Parts  Repair 

•‘0’  level  maint.  PM/CM 

•Computer  based 
training  &  sim 

•Parts  replenishment 

•‘1’  level  Maint.  PM/CM 

•Trainer  site  ops 

•SCM  management 

•‘D’  level  Maint 

•Team  training 

•Maintenance  Mgt 

•Training  management 

°  To  achieve  asset-based 
PBL  -  in  time  metric 
quantity  lessens  but  the 
metric  ‘ quality 9  grows 
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Global  Positioning  System  (GPS) 

Systems  Engineering  (SE) 

Case  Study 


Randy  Bullard 
AFIT/SYA 
25  Oct  2007 
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•  Why  SE  Case  Studies  Are 
Important 

•  Example  -  GPS 

•  Method  of  Analysis 

•  Results 

•  Benefits  (Education  &  Practice) 

•  Foundational  SE  Learning 
Principles 

•  Summary 


W  Why  We  Do  Case  Studies  (<fi> 

- - 

•  AF  Center  for  SE  tasked  to  develop  case  studies 

•  Focus  on  application  of  SE  principles  in  various  programs 

•  Additional  case  studies 

•  Completed: 

•  C-5 

•  F-111 

•  Hubble  Telescope 

•  Theatre  Battle  Management  Core  System  (TBMCS) 

•  B-2 

•  Joint  Air  to  Surface  Standoff  Missile  (JASSM) 

•  In  work:  A-10  &  Peacekeeper 

•  More  on  contract  in  FY08 


Case  Study  Construct 


•  Support  teaching  &  practicing  of  SE  principles 

•  Facilitate  learning  by  emphasizing  long-term  consequences 
of  SE  &  programmatic  decisions  on  program  success 

•  Provide  real-world,  detailed  examples  of  how  SE  process 
attempts  to  balance  cost,  schedule,  &  performance 

•  Used  Friedman-Sage  framework  &  matrix  for  analysis 

•  Identify  learning  principles 

•  Discuss  factors  that  significantly  influenced  successful 
outcome  and  failures  of  the  program 


SE  processes  used  in  today’s  complex  system  and  system-of-systems 
were  matured  and  founded  on  principles  developed  in  the  past. 


Friedman-Sage  Framework 
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•  Developed  by: 

•  Dr  George  Friedman:  University  of  Southern  California 

•  Dr  Andy  Sage:  George  Mason  University 

•  Comprised  of  9  concept  domains  (rows)  &  3 
responsibility  domains  (columns) 

•  Rows  represent  phases  in  SE  life  cycle  &  necessary 
process  and  systems  management  support 

•  Columns  depict  responsibilities  from  both  sides  of  the 
program  (industry  and  government) 

•  Derived  into  matrix 

•  Identifies  learning  principles 

•  Used  as  analysis  baseline 


Friedman-Sage  Matrix 
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Concept  Domain 

Responsibility  Domain 

1.  Contractor 
Responsibility 

2.  Shared 
Responsibility 

3.  Government 
Responsibility 

A. 

Requirements  Definition  and 
Management 

LP  3 

B. 

Systems  Architecture  and 
Conceptual  Design 

C 

System  and  Subsystem  Detailed 
Design  and  Implementation 

D. 

Systems  Integration  and 

Interface 

LP  2 

E. 

Validation  and  Verification 

F. 

Deployment  and  Post 

Deployment 

G. 

Life  Cycle  Support 

H. 

Risk  Assessment  and 

Management 

LP  4 

I. 

System  and  Program 
Management 

LP1 

GPS  Program  Background 
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•  Russians  launched  Sputnik,  1957 

•  Satellite  circled  earth  broadcasting  its  tone 

•  US  engineer  postulated  using  Doppler  Effect 

•  Orbiting  satellite  could  compute  location  on  Earth 

•  Air  Force  &  Navy  established  separate  programs 

•  They  demonstrated  many  key  technologies 

•  DoD  established  Joint  Program  Office  (JPO),  1972 

•  Purpose  was  to  replace  land-based  navigation  aids 

•  Air  Force  was  assigned  to  lead  JPO 

•  Joint  effort  included  Army,  Navy,  &  Coast  Guard 

•  JPO  tasked  to  develop  space-based  navigation  system 


GPS  System 
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System  requirements 

•  Accurate 

•  Global 

•  Real-time 

•  Continuous 

•  Additional  characteristics 

System  design 

•  Space  Vehicle  (SV) 

•  User  Equipment  (UE) 

•  Control  Station  (CS) 


GROUND  ANTENNA 


MONITOR  STATION 


GPS  OV-1 


NAVSTAR  Global  Positioning  System 
Operational  Concept  (OV-1) 


Block  IIF 


Block  IIR-M 


Block  IIA 
Block  MR 


Block  III 


Deter  man  a  lion 


1JTC  Time 
Transfer 


MCS 


Improved  Antejam 


GPS  SE  Approach 


■ 


mm® 


•  JPO  constructed  system  specification 

•  It  became  the  functional  baseline 

•  Strategy  was  to  manage: 

•  Requirements  at  the  performance  level 

•  Interfaces  between  space  vehicles,  control  stations,  &  users 

•  Process  highlighted  cost,  schedule,  &  performance  risks 

•  Impacted  team  derived  alternatives 

•  Decisions  made  quickly  by  management 

•  Program  placed  emphasis  on  staffing  key  positions 

•  JPO  staffed  with  technical  officers  &  civilians 

•  Aerospace  augmented  with  engineering  &  scientific  staff 


Learning  Principles 
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1 .  Programs  must  strive  to  staff  key  positions  with 
domain  experts 

2.  The  systems  integrator  must  rigorously  maintain 
program  baselines 

3.  Achieving  consistent  and  continuous  high-level 
support  and  advocacy  helps  funding  stability, 
which  impacts  SE  stability 

4.  Disciplined  and  appropriate  risk  management 
must  be  applied  throughout  the  lifecycle 


Learning  Principle  1 
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Programs  must  strive  to  staff  key  positions  with 
domain  experts 


•  Program  personnel  were  well-versed  in  their  disciplines 

•  All  possessed  a  systems  view  of  the  program 

•  Entire  team  understood  implications  of  their  work  at  all 
system  levels 

•  They  used  a  knowledge-based  approach  for  decision 
making 

•  Information  was  understood  and  the  base  and  alternative 
solutions  were  accurately  presented. 

•  This  shortened  the  decision  cycle 

•  Additional  benefits  were  realized 

•  Communications  were  better 

•  Working  relationships  were  improved 


Learning  Principle  2 


The  systems  integrator  must  rigorously  maintain 
program  baselines 


•  JPO  retained  the  role  of  managing  and  controlling  the 
systems  specification 

•  This  allowed  control  of  functional  baseline 

•  They  derived  and  constructed  an  “agreed-to”  set  of 
systems  requirements  that  became  the  program  baseline 

•  Performance/Risk/Cost  trade  studies  against  functional 
baseline  used  to  control  both  risk  and  cost 

•  Interface  Control  Working  Group  process  managed  the 
functional  requirements  on  the  allocated  baseline 

•  Processes  gave  JPO  first-hand  knowledge  and  insight  into 
risks  at  lowest  level 


Learning  Principle  3 


Achieving  consistent  and  continuous  high-level 
support  and  advocacy  helps  funding  stability, 
which  impacts  SE  stability 


•  OSD  support  provided  requirements  and  funding  stability 

•  They  provided  advocacy  and  sourced  funding  at  critical  times 

•  They  catalyzed  coordination  among  the  various  services 

•  They  reviewed  &  approved  GPS  JPO  system  requirements 

•  OSD  played  the  central  role  in  the  establishment  and 
survivability  of  the  program 

•  They  had  support  from  Deputy  Secretary  of  Defense 

•  Military  services  were  primary  users  &  eventual  customers 

•  Each  service  initially  advocated  their  individual  programs 

•  SECAF  supplied  manpower  &  facilities 


Learning  Principle  4 


Disciplined  and  appropriate  risk  management  must 
be  applied  throughout  the  lifecycle 


•  GPS  program  structured  to  address  risk  throughout  the 
multiphase  program 

•  Key  risks  were  known  up  front 

•  Contractor  and/or  government  utilized  a  classic  risk 
approach  to  identify  &  analyze  risk 

•  They  developed  and  tracked  mitigation  actions 

•  Various  risks  (design,  manufacturing,  launch)  were 
managed  by  office  who  owned  those  risks 

•  Technical  risks  tracked  by  Technical  Performance 
Measures  (TPMs) 

•  Satellite  weight  &  SLOC  were  tracked 

•  TPMs  addressed  at  weekly  chief  engineer’s  meetings 


SE  Outcomes 
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•  SE  played  major  role  in  GPS  success 

•  Identifying  system  requirements 

•  Integrating  new  technologies 

•  Taking  system  of  systems  approach 

•  Interfacing  with  many  government  &  industry  agencies 

•  Dealing  with  lack  of  an  operational  user  early  in  program 
formation 


•  Key  learning  principles  identify  SE  processes 

•  Application  of  SE  processes  is  required  throughout  life  cycle 

•  Experienced  people  applying  sound  SE  principles, 
practices,  processes,  &  tools  are  necessary  at  each  phase 


GPS  Program  Success 


•  JPO  overcame  numerous  challenges: 

•  Technology,  customers,  organization,  cost,  &  schedule 

•  Integrating  new  technologies 


•  Program  achieved  great  success 

•  Military  relied  upon  extensively 

•  Civilian  applications  growing 

•  Unique  uses  invented 


Imagine  current  technology  without  GPS! 


CSE  Case  Studies 


Case  studies  on  our  website: 
http://www.afit.edu/cse/cases.cfm 


TBMCS 
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Building  Evidence  for  Practice  Selection 
Based  on  Real  Experiences 
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Topics 


Why  do  we  need  a  Clearinghouse? 


■  What  to  expect  in  the  Best  Practices 


Clearinghouse  (BPCh)? 


How  does  the  BPCh  work? 


When  can  I  get  involved? 


Who  to  contact? 
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Why  a  Clearinghouse? 


“Best”  practices  are  recommended,  but... 


Contents 

L  Why  do  we 
^  need  a  BPCh? 

_  What  to  expect 

in  the  BPCh? 

_  How  does  the 

BPCh  work? 


When  can  I 

get  involved? 


Who  to 
contact? 


>Too  many  lists  to  choose  from 

>No  basis  for  selecting  specific  practices 

> Proof  of  effectiveness  is  not  generally  available 

>Not  easy  to  see  connection  between  practices  and 
specific  program  risks  or  issues 

>  Practice’s  success  factors  not  well  understood 

> Resources  are  limited  and  the  return  on  practice 
investment  is  unknown  (costs/benefits) 


> Implementation  guidance  is  inadequate 
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What  are  the  main  requirements? 


Process 

Components 
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Communities 


Roles 
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Central  Repository  for  AT&L  Policy  &  Reference  Materials 


ifjjp  Best  Practice  Giosting 

Connecting  y  ou  to  Govern  merit  and  I  ndustry  Best 


Also  serves  as  the  home  for 
knowledge  gateways  like: 

•  Defense  Acquisition  Guidebook 

•  AT&L  Integrated  Framework 
Chart  (IFC) 

•  Ask  A  Professor 


DoD  &  Industry  Best  Practices 

•  Will  stand  alone  as  a  best 
practices  resource 

•  Will  also  provide  content  for 
CoPs  to  allow  for  additional 
collaboration/input  on  best 
practices 

•  Will  be  included  in  the  enterprise 
search  index/results 


AT&L  ACQuim 

A  value  added  DAU  Search  Se 


Acquisition  Knowledge 
Management  System 


Enterprise  Search  System 

•  Stand-alone  search  and  discovery 
for  AT&L  workforce 

•  Integrated  search  for  AKSS 

•  Searches  open  areas  of  ACC 

•  Integrated  search  for  DAU 
Homepage 

•  Integrated  search  for  DAU 
Intranet 


Collaborative  Tool  for  the  AT&L  Community  Where  the  Workforce 
Contributes  Knowledge  and  Interacts  to  Share  “Know-How” 


Provides  a  nest  of  collaborative 

\  /  \\ 

tools: 

"a!  // 

_ X/ 

•  Communities  of  Practice/Interest 

\  / 

. 

•  Special  Interest  Areas 

•  Limited  Access  Workspaces 

•  DAU  Course  Spaces 

^ . A  / 

Fraunhofer  USA,  Inc 

■■■■■■ 

•  Workflow  Learning  Tools 

•  IFC  Templates 

Center  for  Experl  mental 
Software  Engineering 
Maryland 
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How  do  we  define  a  practice? 

Operational  definition: 

>  A  documented  activity  that  is  described  in  an 
actionable,  repeatable  way 

>  A  description  of  how  to  do  something,  not  a 
general  goal  of  what  to  do 

>  Usable  by  targeted  acquisition  end  users 

>  About  which  we  can  collect  empirical  data  or 
experiences. 


->  may  be  a  process,  method,  or  tool 


Fraunhofer  USA,  Inc 

■  ■■■■■  _ _ _ 


Center  for  Experl  mental 
Software  Engineering 
Maryland 


Slide  6 


©  2007  Fraunhofer  USA 


QOVSnr/Oyy 


United  States 
DEPARTMENT 
of  DEFENSE 


Contents 

Why  do  we 
need  a  BPCh? 

^  What  to  expect 
in  the  BPCh? 

_  How  does  the 

BPCh  work? 

_ When  can  I 

get  involved? 

_  Who  to 

contact? 


What  is  a  practice? 


Distinguished  from: 

>  A  best  practice  area 

❖  ...a  type  of  activity  the  user  can’t  neglect, 
without  specific  advice  on  how  to  do  it.  E.g., 
risk  management 

>  A  lesson  learned 

❖  ...good  advice,  drawn  from  experience, 
without  enough  detail  to  be  clearly 
repeatable.  E.g.,  don’t  overestimate  cost 
savings  from  using  COTS  components. 


Fraunhofer  USA,  Inc 

■■mi _ _ _ 


Center  for  Experl  mental 
Software  Engineering 
Maryland 


Slide  7 


©  2007  Fraunhofer  USA 


QOVSnr/Oyy 


United  States 
DEPARTMENT 
of  DEFENSE 


Contents 

Why  do  we 
need  a  BPCh? 

^  What  to  expect 
in  the  BPCh? 

_  How  does  the 

BPCh  work? 

_ When  can  I 

get  involved? 

_  Who  to 

contact? 


What  is  a  practice? 

For  example: 

■  System  Engineering  Plan: 

>  Risk  Management  Strategy 

Options  to  consider: 

❖COTS  Usage  Risk  Evaluation  (CURE) 
❖Willoughby  templates 
❖SEI's  Taxonomy-based  Risk  Identification 
❖Probability  Consequence  Software 
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What  makes  BPCh  unique? 

■  Not  all  best  practices  are  “best”  for  everybody 

>  Descriptions  of  past  results  in  context,  not  just  what  to  do 

■  Context-sensitive  search 

■  Levels  of  vetting  of  content 

■  Subject  Matter  Experts  as  practice  owners 

■  Pointers  to  existing  sites,  resources,  examples 
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What  are  the  main  process  steps? 


Name:  Practice  X 


•Practice  X  has  been  successfully  applied  ... 
•Use  It  to  . . . 

•For  more  information  click  on  the  following  links: 


Practice 

Maturity 


Source 

Context 

Results 


Center  for  Experimental 
Software  Engineering 
Maryland 


Source  Source 

Context  Context 

Results  Results 
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What  are  examples  of  content  sources? 


Systems  Engineering 

>  OUSD  /  SEP  Review  Team 

>  OUSD  /  PSR  Teams  and  Systemic  Analysis 

>  Experience  reports  from  NDIA-SE  &  similar 

■  Software  engineering 

>  ARDEC’s  Software  Enterprise  /  Picatinny  Arsenal 

■  Acquisition 

>  NDIA 

■  Other  ideas? 

>  Existing  DOD  guidebooks  /  standards 

>  Existing  best  practice  /  lessons  learned  sites 

>  Expert  interviews 

>  Conference  presentations,  experience  reports 
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What  are  some  example  practices? 

■  Requirements  analysis 

>  Govt-only  review  of  RFP 

>  Distribute  requirements  database  for  bidders 

■  Reporting  /  stakeholder  communication 

>  Establish  a  battle  rhythm  for  meetings  in  SEP  -  what 
gets  done  daily,  weekly,  semi-annually. 

■  Interfaces 

>  PEO-level  coordination 

■  General... 

>  Independent  technical  reviews 

>  Integrated  data  environment 
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What  is  the  BPCh  content  pedigree? 


■  Pedigree  comes  from  information  that  is 
available  on  each  piece  of  evidence: 

>  Target  role  (acquirer,  developer) 

>  Domain  (warfighter,  business,  intelligence,  enterprise 
integration  environment) 

>  Criticality  level  (normal,  mission,  safety,  security) 

>  Integration  level  (software  application,  standalone 
subsystem,  platforms,  major  system,  system  of  systems) 

>  Environment  (military,  other  govt.,  industry,  academia) 

>  ACAT  level  (I,  IA,  II,  III) 

>  Lifecycle  phases  where  practice  used:  (Concept 
refinement,  Technology  development,  System 
development  &  demonstration,  etc.) 

>  Organizational  scope  (individual,  project,  program, 
organization,  enterprise) 
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How  do  we  classify  “trustabi I ity”? 


Evidence  is  scored  based  on  objective  measures: 


Total  = 

How 

practice 

applied 

How 

results 

measured 

How 

reported 

Who 

reported 

[0..20] 

[0..7] 

[0..5] 

[0..5] 

[0..3] 

■  Practices  are  described  as  a  sum  of  evidences 
with  different  ratings: 
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Practice  detail 


Practice  Name:  Inspections 

I  a  well-defined  review  process  for  finding  and  fixing  defects  in  work  products  from  all  phases  of 


Description: 

Practice 

Status: 


development. 

Summarized  -  waiting  for  vetting 


Resources: 

DAU's  Tech  Review  Course 


Description: 


Summary 

Wh<  T 

9et '  Primary  Benefit: 

_  wh<  The  majority  of  sources  show  that  inspections  have  a  measurable  impact  on  the 

coni  ■  ■ 

number  of  defects  taken  out  of  software  documents. 


Inspections  have  a  better  ROI  for  the  effort  spent  performing  them,  than  other  common  means  of  defect 
reduction  like  testing.  I 

All  sources  agree  that  inspections  are  useful  within  the  system  development  phase,  Multiple  authors  discuss  that  inspections  are  usable 
in  all  phases  of  software  development  although  more  benefits  accrue  when  they  are  used  earlier  in  the  lifecycle. 

Organizational  Scope: 
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Summary 


Primary  Benefit: 

The  most  often- mentioned  impact  is  on  quality:  The  majority  of  sources  show  that  inspections  have  a  measurable  impact  on  the  number 
of  defects  taken  out  of  software  documents.  As  a  side-effect  some  sources  also  argue  that  they  lead  to  fewer  defects  being  generated 
in  the  future,  through  skill  development  of  the  personnel  involved. 

A  secondary  benefit  is  on  cost:  Inspections  have  a  better  ROI  for  the  effort  spent  performing  them,  than  other  common  means  of  defect 
reduction  like  testing. 

Life  Cycle  Phases: 

All  sources  agree  that  inspections  are  useful  within  the  system  development  phase.  Multiple  authors  discuss  that  inspections  are  usable 
in  all  phases  of  software  development,  although  more  benefits  accrue  when  they  are  used  earlier  in  the  lifecycle. 

Organizational  Scope: 

While  there  are  numerous  benefits  that  can  be  achieved  within  a  project,  authors  also  described  an  overall  advantage  to  the  entire 
organization  or  a  given  program  that  can  result  from  implementation  of  inspections  across  projects. 

Primary  Target: 

Almost  all  sources  focus  on  the  use  of  inspection  within  the  development  organization  (to  minimize  their  test  and  rework  effort). 

However,  the  technical  review  process  at  NavAir  can  be  considered  a  variant.  It  has  been  used  across  many  projects  by  acquirers  to  help 
monitor  the  developers  they  are  overseeing. 


Barriers: 

The  primary  inhibitors  seem  to  be  in  the  realm  of  developer  motivation: 

-  Inspections  are  perceived  as  being  labor  intensive  in  nature 

-  Payback  is  delayed  (i.e.  benefits  are  not  seen  until  long  after  foe  effort  is  spent) 

than  the  team  to  coordinate  the  process,  and  personnel  who  provide  support  for  measurement  and  interpretation. 

Inspections  may  not  be  put  into  common  use  if  there  is  no  way  to  allow  some  customization  to  the  division/group. 

Madachy  argues  that,  since  there  is  some  overhead  cost  involved  in  inspections,  projects  that  already  have  extremely  low  defect  rates 
(e.g.  development  using  the  Cleanroom  paradigm)  may  not  see  a  cost-benefit. 

Enablers: 

There  is  a  need  for  local  champions  and  management  support.  Development  teams  who  implement  inspections  need  trained  moderators 
and  support  materials.  Providing  explicit  training  helps  improve  effectiveness. 

Inspections  work  best  if  management  is  not  present  at  inspections,  to  remove  the  threat  that  defect  information  could  be  used  to 
evaluate  workers. 

Net  Impact  on  Cost: 
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List  of  evidences  for  this  practice 

Rating  Status 

Evidence  Name 

Created  on 

20 

Included  in 
summary 

Advances  in  Software  Inspections 

2/16/20D5 

19 

Included  in 
summary 

An  Analvsis  of  Defect  Densities  Found  Durina  Software  Inspections 

1/13/2005 

19 

Included  in 
summary 

Measurina  Inspections  at  Litton 

1/12/2005 

IS 

Included  in 
summary 

Experience  with  Inspection  in  Ultra laroe-Scale  Developments 

1/13/2005 

17 

Included  in 
summary 

Kev  Lessons  in  Achievina  Widespread  Inspection  Use 

1/13/2005 

17 

Included  in 

Space  Shuttle  Primary  Onboard  Software  Development:  Process  Control  and  Defect  Cause 

2/16/2005 

summary 

Analvsis 

15 

Included  in 
summary 

Comparina  the  Effectiveness  of  Software  Testina  Strateaies 

2/16/2005 

13 

Included  in 
summary 

Report  on  the  Loss  of  the  Mars  Climate  Orbiter  Mission 

2/17/2005 

13 

Included  in 
summary 

The  Empirical  Investiaation  of  Perspective-Based  Readina 

2/16/2005 

12 

Included  in 

Applvina  Proaram  Comprehension  Techniques  to  Improve  Software  Inspections 

1/13/2005 

summary 

12 

Included  in 
summary 

NavAir  technical  reviews 

2/17/2005 

S 

Included  in 
summary 

What  We  Have  Learned  about  Fiahtina  Defects 

1/13/2005 
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Is  there  any  evidence  that  inspections  will 
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Sources: 

Madachy,  Ray 


Project  detail 


Contents 

Why  do  we 
need  a  BPGh? 


Affiliations: 

Litton 


Domain: 

Environment: 


_  What  to  expect 

in  the  BPCh? 


How  does  the 
BPCh  work? 


When  can  I 

get  involved? 

Who  to 

contact? 


Name: 

Criticality  Level 
Integration  Lev 
Domain: 
Environment: 
Applied  how: 
Total  System  cc 
ACAT  Level: 


Applied  how; 

Total  System  cost: 
ACAT  Level: 

Team  Size 

Team  Environment: 


Team  Size 


Team  Environmi 

Life  Cycle  Phases: 


Life  Cycle  Phase 


Business 

Industry 

Multiple  production  projects 


System  Development  &  Demonstration  i 


Fraunl 

■  _ 

■  ■■■■  Center  foi 
I  Software 
I  Maryland 


Organizational  j  Organizational  scope:  Organization 

Target  Role: 

Target  Role:  Developer 


Primary  Benefit: 


hUzJSUIL 


ny 


recur  r  ir  r  ler  luatiur  is 


Improve  Quality 


Net  Impact  on  Cost:  Reduced  cost 

p6:  “Our  project  director  set  goals  to  save  at  least  50%  of  integration  effort../'  which  are  being  met  using  inspections. 

p9:  “The  net  return  per  inspection...  range  from  64  to  200  person-hours  saved  per  inspection..." 

Cost  involves  extra  effort  in  addition  to  meeting  time.  p9:  'Our  results  concur  with  [Grady92]  and  other  references  that  suggest  a  large 
proportion  of  preparation  time  to  meeting  time..." 


Resulting  recommendations 

Primary  Benefit:  Improve  Quality 

Barriers: 

pl3:  “We  have  also  found  that  the  use  of  an  SEPG  peer  review  coordinator  goes  a  long  way  to  keep  the  process  intact."  That  is, 
sustaining  requires  someone  at  a  higher  level  responsible  for  coordinating  the  practice. 

nl^1  ^A^hilp  inQnpr1~inn  jnrl  nrprurjfinn  pffnrt~  Qtjw  fairlw  rnnQtjnt~  th p  rpwnrlf  jnrl  tpvt  prrnr  fivinn  pffhrt  warw  with  th p  rlpfprt  rptpQ  [  1 


Net  Impact  on  Cost 

•  During  design  and  coding,  about  3%  of  the  total 
project  effort  was  used  for  inspection 

•  Hie  net  return  per  inspection  range  from  64  to  200 
person- hours  saved  per  inspection 

p/:  ""An  overall  inspection  efficiency  of  oil I  hat  is,  inspections  removed  approximately  3U%  or  detects  in  trie  inspected  products, 
improving  quality. 

p6:  "...achieved  significant  results  in  terms  of  defect  reduction  and  return  on  investment" 

p9:  “Besides  the  detection  of  errors  during  inspection,  less  errors  are  originally  generated  due  to  authors  being  more  careful  as 
inspection  metrics  are  publicized." 

p9:  “When  comparing  trouble  report  data  before  and  after  inspections  were  introduced,  there  is  about  a  2/3  reduction  in  trouble  report 
density  during  integration..." 
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List  of  evidences  for  this  practice 

Rating  Status 

Evidence  Name 

Created  on 

20 

Included  in 
summary 

Advances  in  Software  Inspections 

2/16/2005 

19 

Included  in 
summary 

An  Analysis  of  Defect  Densities  Found  Durina  Software  Inspections 

1/13/2005 

19 

Included  in 
summary 

Measurina  Inspections  at  Litton 

1/12/2005 

IS 

Included  in 
summary 

Experience  with  Inspection  in  Ultra larae-Scale  Developments 

1/13/2005 

17 

Included  in 
summary 

Kev  Lessons  in  Achievina  Widespread  Inspection  Use 

1/13/2005 

17 

Included  in 

Space  Shuttle  Primary  Onboard  Software  Development:  Process  Control  and  Defect  Cause 

2/16/2005 

summary 

Analysis 

15 

Included  in 
summary 

Comparina  the  Effectiveness  of  Software  Testina  Strateaies 

2/16/2005 

13 

Included  in 
summary 

Report  on  the  Loss  of  the  Mars  Climate  Orbiter  Mission 

2/17/2005 

13 

Included  in 
summary 

The  Empirical  Investiaation  of  Perspective-Based  Readina 

2/16/2005 

12 

Included  in 

Applvina  Proaram  Comprehension  Techniques  to  Improve  Software  Inspections 

1/13/2005 

summary 

12 

Included  in 
summary 

NavAir  technical  reviews 

2/17/2005 

S 

Included  in 
summary 

What  We  Have  Learned  about  Fiahtina  Defects 

1/13/2005 
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Is  there  any  evidence  that  inspections  will 
actually  save  us  time  and  money? 


Primary  Benefit: 
Barriers: 


Resulting  recommendations 

Improve  Quality 


Lack  of  training  can  be  a  barrier:  p.  74S:  "Some  organizations  have  started  inspections  without  proper  education  and  have  achieved 
so  me  success;  but  less  than  others  who  prepared  their  participants  fully.  This  has  caused  some  amount  of  start-over,  which  was 


ffus' 


p.  7 
B) 

-  PR 

-  Exi 


Ena 

p.  7- 


Neec 

askir 


Net  Impact  on  Cosh 

Inspections  have  the  effect  of  slightly  front- 
end  loading  the  c  ommitment  of  resourc es, 
adding  to  requirements  and  design 


perl 


P.  7 

than 


p.  7 


p.  7 
deve 
code 


In  eac  h  instance,  the  new  uses  of  inspection 
were  found  to  improve  product  quality  and  to 
be  cost  effective,  i.e.,  it  saved  more  than  it 


en 

Dr 


ire 


nd 


Net  Impact  on  Quality: 


Increased  quality 


p.  744:  "In  each  instance,  the  new  uses  of  inspection  were  found  to  improve  product  quality  and  to  be  cost  effective,  i.e.,  it  saved  more 
than  it  cost. 11 


p.  744:  Fagan  quotes  an  IBM  director  saying:  "Since  we  introduced  the  inspection  process  in  1974,  we  have  achieved  significant 
improvements  in  quality.  IBM  has  nearly  doubled  the  number  of  lines  of  code  shipped...  since  1976,  while  the  number  of  defects  per 
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What  is  the  status  of  BPCh? 


■  Run  by  Defense  Acquisition  University  (DAU) 

>  An  OUSD(AT&L)  training  institution 

■  IT  Components: 

>  BPCh  vl.O  developed,  undergoing  usability  training 

■  Processes: 

>  Piloted  for  submitting  practices  and  evidence,  e.g. 

❖  Integration  with  DAU  traditional  and  e-classrooms 
❖Solicited  via  the  tool 

■  Roles  /  Communities: 

>  SMEs  from  DAU,  Services,  Agencies  and  Industry  are 
being  engaged 

>  Practice  Provider  Network  being  forged  to  create  a  list  of 
trusted  content  sources 


Public  debut  of  BPCh  vl.O  planned  for  Spring  2008 
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How  are  practices  prioritized? 


■  Priorities  are  set  by  Content  Advisory  Group 

>  Periodic  meetings  to  review  content,  recommend  areas  of 
interest 

♦Mow-hanging  fruit  or  areas  of  high  concern 

>  Also  to  review  opportunities  to  share  content  with  other 
best  practice  /  lessons  learned  initiatives 

❖  Looking  for  speakers! 

>  Chaired  by  David  Castellano,  OUSD  (AT&L) 

>  Recommendations  executed  by  Content  Manager, 

Forrest  Shull 


>  Current  hot  topics  include:  Risk  management,  Earned 
Value  Management,  Requirements  engineering 

->  Priorities  constantly  reviewed  and  updated 


Fraunhofer  USA,  Inc 

■  ■■■■■  _ _ _ 


Center  for  Experl  mental 
Software  Engineering 
Maryland 


Slide  23 


©  2007  Fraunhofer  USA 


„^SITfOA, 


United  States 
DEPARTMENT 
of  DEFENSE 


Contents 

Why  do  we 
need  a  BPCh? 

_  What  to  expect 

in  the  BPCh? 

_  How  does  the 

BPCh  work? 

^  When  can  I 

get  involved? 

_  Who  to 

contact? 


Can  I  suggest  content? 

■  YES! 

>  We  are  looking  for  practice  suggestions  to 
ensure  the  usefulness  of  the  BPCh  to  the  user 
community 

>  We  are  looking  for  evidence  to  add  to  an 
existing  practice 

>  Everyone  can  suggest  practices 
->  simply  e-mail  us 


->  please  fill  in  the  survey  that  we  circulate 
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How  can  I  participate  next  year? 


■  Visit:  https://acc.dau.mil/bpch 

■  Built-in  feedback  forms  in  the  application 

>  ...To  give  us  a  lead 

>  ...To  suggest  a  practice  we  should  have 

>  ...To  tell  us  your  experience  with  a  practice 

>  ...To  give  us  a  detailed  experience  report 

■  Ability  to  integrate  BPCh  with  in-house 
best  practice  /  lessons  learned  systems 

■  Elicitation  workshops 

>  Send  us  your  suggestions 
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Questions? 
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Forrest  Shull 
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List  of  used  abbreviations 


ACC: 

Acquisition  Community  Connection 

ACAT: 

Acquisition  CATegory 

AKSS: 

AT&L  Knowledge  Sharing  System 

AT&L: 

Acquisition,  Technology  and  Logistics 

BPCh: 

(Acquisition)  Best  Practices  Clearinghouse 

CoP: 

Communities  of  Practice 

COTS: 

Components  Off  The  Shelf 

DAU: 

Defense  Acquisition  University 

DoD: 

U.S.  Department  of  Defense 

IFC: 

Integrated  Framework  Chart 

MOSS: 

Microsoft  Office  SharePoint  Server 

OSD: 

Office  of  the  Under  Secretary  of  Defense 

ROI: 

Return  On  Investment 

SAM: 

Software  Acquisition  Management 

SE: 

Systems  Engineering 

SEI: 

Software  Engineering  Institute 

SMEs: 

Subject  Matter  Experts 
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Understanding  Social  Networks-  A  Key 
Requirement  for  System  Engineers 


Mr.  Karl  Selke 

Systems  Engineering  Analyst 

Evidence  Based  Research,  Inc 
1595  Spring  Hill  Road  Suite  250 
Vienna  VA  22182  (703)-893-6800 

selke@ebrinc.com 


Agenda 


•  Introduction 

•  Wicked  Problems 

•  Understanding  Social  Networks 

•  Social  Network  Analysis  Process 

•  Concluding  Thoughts 


My  motivation:  Preliminary  research  for  an  Army  Small  Business  Innovative  Research 
Solicitation 

-  Social  Networks  and  Organizations  (2003)  by  Martin  Kilduff  and  Wenpin  Tsai 

-  Social  Network  Analysis:  Methods  and  Applications  (1994)  by  Wasserman  and 
Faust 

My  question:  Is  quantification  and  visualization  of  the  maze  of  social  interactions 
important  for  project  success? 

My  underlying  assertion:  Technical  success,  enabled  by  good  systems  engineering, 
should  be  accompanied  by  successful  engineering  of  the  social  network  in  which  the 
project  exists. 

My  underlying  mission:  Utilize  computational  social  sciences’  approaches  for 
anticipating  and  managing  the  complexity,  inherent  in  diverse,  potentially  conflicting 
stakeholders,  project  teams,  and  project  managers. 


The  social  network  matters  and  the  following  is  merely  a  simple 
articulation  of  the  problem  and  a  potential  solution. 


The  Importance  of  “Wicked  Problem”  on  Systems 
Engineering 


\ 

/ 


Every  textbook  of  system  engineering  starts  with  an  enumeration  of  these  phases: 
“understand  the  problems  or  the  mission,  ”  “gather  information,  ”  “work  out  solution,  ”  or 
the  like.  For  wicked  problems,  however,  this  type  of  Schema  will  not  work. 

•  Rittel  and  Webber’s  paper  introduced  the  concept  of  “wicked”  problems  to  explain 
why  scientific  approaches  and  methods  for  solving  societal  problems  are  inherently 
flawed. 

•  As  systems  engineers  have  turned  proper  prior  planning  into  a  science,  this  raises 
the  question  as  to  the  distribution  of  “wickedness”  faced  by  systems  engineers  and  if 
so,  whether  the  field  of  systems  engineering  has  adequately  prepared  system 
engineers  to  address  them. 


Rittel  and  Weber,  Dilemmas  in  a  General  Theory  of  Planning  (1973) 


The  social  network  matters  and  the  following  is  merely  a  simple 
articulation  of  the  problem  and  a  potential  solution. 


Properties  of  Wicked  Problems 


♦ 


The  ten  properties  are  as  follows: 

1 .  There  is  no  definitive  formulation  of  a  wicked  problem. 

2.  Wicked  problems  have  no  stopping  rule. 

3.  Solutions  to  wicked  problems  are  not  true-or-false,  but  good-or-bad. 

4.  There  is  no  immediate  and  no  ultimate  test  of  a  solution  to  a  wicked  problem. 

5.  Every  solution  to  a  wicked  problem  is  a  “one-shot  operation”;  because  there  is  no 
opportunity  to  learn  by  trial-and-error,  every  attempt  counts  significantly. 

6.  Wicked  problems  do  not  have  an  enumerable  (or  an  exhaustively  describable)  set  of 
potentia  solutions,  nor  is  there  a  well-described  set  of  permissible  operations  that 
may  be  incorporated  into  the  plan. 

7.  Every  wicked  problem  is  essentially  unique. 

8.  Every  wicked  problem  can  be  considered  to  be  a  symptom  of  another  problem. 

9.  The  existence  of  a  discrepancy  representing  a  wicked  problem  can  be  explained  in 
numerous  ways.  The  choice  of  explanation  determines  the  nature  of  the  problem’s 
resolution. 

10.  The  Planner  has  no  right  to  be  wrong. 


Ml  For  further  articulation  of  these  properties,  one  can  refer  to  Rittle  and  Webber,  Dilemmas  in  a  General  Theory  of 
Planning  (1973). 


One  can  define  “wickedness”  as  the  extent  to  which  one  or  more  of 

these  properties  are  dominant. 


EB 


“Wicked”  System  Engineering  Problems 


Establish  good  relationships  and  open  communications  between  systems  engineers 
and  stakeholders.  This  is  helpful  when  negotiations  begin  to  refine  and  clarify  the  set 
of  requirements. 

•  In  general,  project  managers  and  systems  engineers  work  to  meet  project 
performance  requirements  within  planned  schedule  and  cost. 

•  There  are  a  set  of  observable  reasons  that  are  asserted  to  be  the  most  common 
rationales  for  why  project  objectives  are  not  met. 

-  Inadequate  articulation  of  requirements 

-  Poor  planning 

-  Inadequate  technical  skills  and  continuity 

-  Lack  of  teamwork 

-  Poor  communications  and  coordination 

-  Insufficient  monitoring  of  progress 

-  Inferior  corporate  support 

INCOSE  Systems  Engineering  Handbook 

Howard  Eisner,  Essentials  of  Project  and  Systems  Engineering  Management  (2002)  page  9 


It  is  more  useful  to  view  these  rationales  as  problem  areas. 


•  Requirement  Definition-  customer  defined  requirements  for  the  system. 

•  Project  Planning-  initial  project  plan,  personnel  plans,  and  subsequent  updates  over 
time. 

•  Necessary  Personnel  Resources-  required  technical  staff 

•  Collaboration-  ability  for  personnel  to  function  as  a  “team” 

•  Interaction  Network-  propagation  and  instantiation  of  guidance  or  orders 

•  Management  Approach-  leadership’s  guiding  principles  of  interaction  with  personnel 

•  Organizational  Environments-  external  (to  the  project  team)  networks  containing  all 
the  external  factors  that  are  relevant  to  the  project. 


Underlying  each  of  these  problem  areas  are  complex  interactions 
amongst  the  internal  project  personnel,  external  project  personnel, 

customers,  and  stakeholders. 


“Wicked”  Engineering  Management  Problems-  continued 


Requirement  Definition 


Social  Interactions 


Given  that  the  problems  are  heavily  correlated  to  the  outcomes  of  social 
interactions,  it  suggests  that  these  problems  may  have  “wicked” 

aspects. 


Requirement  Definition 


♦ 


Properties  of  Wicked  Problems 

The  ten  properties  are  as  follows: 

1.  There  is  no  definitive  formulation  of  a  wicked  problem. 

2.  Wicked  problems  have  no  stopping  rule. 

3.  Solutions  to  wicked  problems  are  not  true-or-false,  but  good-or-bad. 

4.  There  is  no  immediate  and  no  ultimate  test  of  a  solution  to  a  wicked  problem. 

5.  Every  solution  to  a  wicked  problem  is  a  “one-shot  operation”;  because  there  is 
no  opportunity  to  learn  by  trial-and-error,  every  attempt  counts  significantly. 

6.  Wicked  problems  do  not  have  an  enumerable  (or  an  exhaustively  describable) 
set  of  potential  solutions,  nor  is  there  a  well-described  set  of  permissible 
operations  that  may  be  incorporated  into  the  plan. 

7.  Every  wicked  problem  is  essentially  unique. 

8.  Every  wicked  problem  can  be  considered  to  be  a  symptom  of  another  problem. 

9.  The  existence  of  a  discrepancy  representing  a  wicked  problem  can  be  explained  in 
numerous  ways.  The  choice  of  explanation  determines  the  nature  of  the  problem’s 
resolution. 

10.  The  Planner  has  no  right  to  be  wrong. 


Ml  For  further  articulation  of  these  properties,  one  can  refer  to  Rittle  and  Webber,  Dilemmas  in  a  General  Theory  of 
Planning  (1973). 


Property  7  can  dominant  the  interaction  of  customers  and  systems 

developers 


Growing  Complexity  of  Social  Interactions 


♦ 
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m  Renee  Stevens,  Engineering 
Enterprise  Systems:  Challenges 
and  Prospects 


Unfortunately  for  systems  engineering,  understanding  the  situation  (or 
the  social  complexity)  is  the  wicked  problem. 


Understanding  Social  Networks 


System  Thinking 

•  “The  systems  engineering  perspective  is  based  on  systems  thinking.  Systems 
thinking  occurs  through  discovery,  learning,  diagnosis,  and  dialog  that  lead  to 
sensing,  modeling,  and  talking  about  the  real-world  to  better  understand,  define,  and 
work  with  systems.”^ 

•  Systems  Thinking  is  an  important  component  of  systems  engineering — it  may  actually 
be  the  defining  characteristic  that  separate  great  engineers  from  good  engineers. 

“Big  Picture”Knowledqe 

•  “Big  Picture”  Knowledge  is  simply  the  knowledge  of  the  internal  and  external 
environment  including  communities  of  interest,  processes,  and  specific  players. 

•  Big  Picture  Knowledge  is  a  necessary  component  of  effective  program  management. 


Ml  INCOSE  Systems  Engineering  Handbook 


Is  there  a  collaboration  mechanism  that  combines  “gray  beard” 
knowledge  of  the  big  picture  with  the  systems  thinking  capability  of 

systems  engineers? 


Collaboration  Mechanism 


♦ 


Systems  Thinking 
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This  interaction  between  the  suave  business  knowledge  program  manager  and 
the  experienced  systems  thinker  as  the  chief  systems  engineer  may  mitigate 
“wickedness.” 


Players  Comprising  the  Social  Network 


♦ 


Is  it  possible  to  institutionalize  knowledge  of  the  social  network? 


[SEE 


Brief  Description  of  Social  Network  Analysis 


♦ 


•  “Social  network  analysis  is  based  on  an  assumption  of  the  importance  of  relationships 
among  interacting  units.  The  social  network  perspective  encompasses  theories, 
models,  and  applications  that  are  expressed  in  terms  of  relational  concepts  or 
processes.” 

•  Social  network  analysis  should  not  be  outside  the  capability  of  a  systems  engineer, 
but  rather  its  benefits  would  fold  nicely  into  mission  engineering — a  core  component 
of  systems  engineering. 

-  Mission  engineering  is  that  often  overlooked  aspect  in  which  the  system 
developers  ask  the  ‘big  picture’  question:  “why  is  this  system  being 
developed?”!^ 

-  Social  network  analysis  makes  the  question  more  fundamental:  “Why  is  this 
system  being  developed  and  who  is  important  to  its  sustained  success?” 

•  “The  purpose  of  social  network  analysis  is  to  provide  insightful  information  and 
inferences  on  the  organization  and  structural  properties  of  a  network,  given  its  nodes 
and  relations. J 


Wasserman  and  Faust,  Social  Network  Analysis:  Methods  and  Applications  (1994)  page  10 
Howard  Eisner,  Essentials  of  Project  and  Systems  Engineering  Management  (2002)  page  195 

Dr.  Cioffi-Revilla  and  Dr.  Sean  Obrien,  Computational  Analysis  in  US  Foreign  and  Defense  Policy 
(2007) 


Social  Network  Analysis  Process 


♦ 
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Continuous  process  where  individuals,  organizations,  and  relationships  should  be 
systematically  identified,  processed,  and  refined  into  actionable  information 


Social  Network  Analysis  Process-  Inputs 


♦ 


•  The  inputs  for  this  process  are  the  data  and  information  describing  the  individuals, 
organizations,  and  relationships  comprising  both  the  internal  and  external  system 
engineering  and  management  environment. 

-  The  nature  of  the  data  can  include  everything  from  friendship  ties  and 
organization  memberships  to  personality. 

-  This  data  often  resides  with  numerous  individuals;  however,  the  project  manager 
will  likely  have  a  solid  foundation. 

-  The  data  can  be  collected  in  several  ways  including  roster  type  surveys, 
egocentric  data  from  interviewing  available  people,  archival  data  such  as 
employee  records  based  on  the  barriers  of  access. 

•  Depending  on  the  scope  of  the  project,  the  following  entities  may  be  relevant: 

-  Stakeholders  including  customers 

-  Project  Personnel 

-  Corporate/Government  Leadership 

-  Support  Organizations 

-  Collaborative  Networks  (fellow  systems  engineers,  project  managers,  etc) 

-  Communities  of  Interest  (similar  projects,  cutting-edge  technical  expertise) 


Ml  Howard  Eisner,  Essentials  of  Project  and  Systems  Engineering  Management  (2002)  page  9-30 
[11  Wasserman  and  Faust,  Social  Network  Analysis:  Methods  and  Applications  (1994)  page  28-66 


Social  Network  Analysis  Process-  Constraints 


♦ 


•  There  is  a  strong  argument  that  advocates  progress  over  process  which  would 
stringently  reject  additional  action  items  for  systems  engineers  to  accomplish. 

-  The  time  pressure  for  project  managers  and  system  engineers  is  often 
overwhelming. 

-  There  may  also  be  reluctance  amongst  system  engineers  to  capture  and  codify 
individual  and  relationship  data  as  part  of  the  system  engineering  process,  but 
rather  let  the  data  reside  in  their  own  mental  models. 

-  Thus,  both  time  pressure  and  individual  inertia  will  constrain  this  process. 

•  Data  collection  and  a  systematic  examination  into  the  social  network  to  prevent  the 
eschewing  of  project  success  by  key  stakeholders  present  several  challenges  such 
as  personally  sensitive  information. 


Social  Network  Analysis  Process-  Enablers 


•  Social  network  analysis  as  a  component  of  the  broader  field  of  computational  social 
science  has  spawned  a  great  deal  of  research  in  the  past  ten  years. 

-  This  has  yielded  social  network  software  packages  such  as  UCINET  and 
visualization  software  such  as  KrackPlot  and  Netdraw  (there  are  a  good  many 
more  tools). 

•  The  automation  of  certain  types  of  analysis  such  as  deducing  centrality  of  an 
individual  greatly  empowers  social  network  appliers  rather  than  only  academics. 

•  There  are  current  initiatives  in  the  Department  of  Defense  seeking  to  utilize  social 
network  analysis  related  to  modeling  the  performance  of  joint,  interagency,  multi- 
organizational,  and  multinational  organizations. 

•  Another  enabler  is  the  systems  thinkers,  particularly  the  graybeard  contractors,  who 
have  spent  forty  years  in  the  business  and  have  deep  understandings  of  the  social 
networks. 


[1]  See  Wasserman  and  Faust  or  Kilduff  and  Tsai  for  more  information  concerning  tools. 


Social  Network  Analysis  Process-  Analysis 


♦ 


The  network  of  relationships  within  which  we  are  embedded  may  have  important 
consequences  for  the  success  or  failure  of  our  projects.  Evidence  suggests  that  the 
types  of  networks  we  form  around  ourselves  affect  everything  from  our  health,  to  our 

career  success,  to  our  very  identities. 

•  There  are  numerous  activities  that  can  be  done  within  the  field  of  social  network 
analysis  that  could  be  useful  for  systems  engineers. 

-  Centrality  Analysis:  Analyzes  the  data  based  on  several  indices  to  determine  the 
degree  to  which  the  network  is  centered  around  a  few  key  individuals. 

-  Clique  Analysis:  Determines  how  many,  what  kind,  and  other  clique  related 
information  that  may  be  useful  to  the  analyst. 

-  Social  Structure  Analysis:  Provides  a  delineation  of  structural  features  not 
observable  by  inspection. 

-  Blockmodeling:  Aims  to  discover  groups  in  which  there  are  no  or  limited 
connections. 

•  The  traditional  approach  for  performing  this  type  of  analysis  is  to  trust  the  insights  of 
the  retiring  graybeards  in  whom  past  experience  has  grown  wisdom  and  connections. 

[11  Kilduff  and  Tsai,  Social  Networks  and  Organizations  (2003)  page  2. 

[21  Kilduff  and  Tsai,  Social  Networks  and  Organizations  (2003) 


The  potential  knowledge  gleaned  from  these  activities  can  provide  actionable 
information  to  project  managers  and  systems  engineers  to  provide  assurance  for 
project  success  given  technical  success. 


Social  Network  Analysis  Process-  Output 


The  analysis  provides  an  increasing  understanding  of  the  social  network  in  which  the 
project  exists. 

Social  network  analysis  yields  actionable  information  vital  for  decision-making. 

Four  useful  features  of  social  network  analysis: 

-  Social  Capital 

-  Embeddedness 

-  Network  Centrality 

-  Structural  Holes. 


Output-  Understanding  Social  Networks:  Social  Capital 


♦ 


All  leadership  is  influence  -  John  C.  Maxwell,  Injoy,  Inc. 

•  Social  capital  can  be  defined  as  “the  benefits  that  accrue  to  the  collectivity  as  a  result 
of  the  maintenance  of  positive  relations  between  different  groups,  organizational 
units,  or  hierarchical  levels.  At  the  individual  level,  this  concept  can  be  defined  as  the 
potential  resources  inherent  in  an  actor’s  set  of  social  ties.” 

•  Echoing  this  concept,  Clark  Aldrich  in  Simulations  and  the  Future  of  Learning  writes 
that  “there  is  a  currency  that  all  people  earn  and  spend  called  personal  influence.” 

•  Unfortunately,  not  all  program  managers  and  systems  engineers  have  strong  social 
networks  that  yield  large  amounts  of  influence. 

•  Social  network  analysis  can  be  combined  into  the  project’s  personnel  plan  to 
intelligently  spend  your  team’s  social  capital  and  your  own  personal  influence  paying 
special  attention  to  the  dynamics  which  could  lead  to  unintended  consequences. 


[11  Kilduff  and  Tsai,  Social  Networks  and  Organizations  (2003)  page  28 
[2]  (2004)  page  161 


Output-Understanding  Social  Networks:  Network 


Centrality 


♦ 


Richard  knew  that  if  the  quality  program  was  going  to  get  anyplace  in  the  organization  he 
would  have  to  place  on  the  support  of  the  princes  and  to  dance  past  the  kings  - 

David  T.  Kearns dl 

•  Network  centrality  is  certainly  implicit  when  discussing  social  capital,  embeddedness, 
and  structural  holes. 

•  Actors  can  be  central  in  the  network  from  a  variety  of  different  perspectives  such  as 
popularity,  go-between  ties,  or  proximity  to  popular  individuals. 

-  Ascertaining  this  can  be  very  informative  because  an  actor  with  a  high  centrality 
level  is  in  “direct  contact  or  is  adjacent  to  many  other  actors.” 

-  “This  actor  should  then  begin  to  be  recognized  by  others  as  a  major  channel  of 
relational  information,  indeed,  a  crucial  cog  in  the  network,  occupying  a  central 
location.” 


[11  Prophets  in  the  Dark:  How  Xerox  Reinvented  Itself  and  Beat  Back  the  Japanese  page  278. 

[21  Wasserman  and  Faust,  Social  Network  Analysis:  Methods  and  Applications  (1994)  page  178-179 


Output-  Understanding  Social  Networks:  Embeddedness  ♦ 


In  Washington,  as  elsewhere,  power  does  not  always  follow  organizational  charts;  a 
person’s  title  does  not  necessarily  reflect  the  power  that  he  or  she  has. 

•  The  concept  of  embeddedness  purports  the  notion  that  work-related  transactions 
tend  to  overlap  with  patterns  of  social  relations. 

-  This  is  very  similar  to  the  “Good  Old  Boy  System” 

-  This  has  broad  consequences  in  information  flow  and  collaboration  in  the 
professional  environment. 

•  Discovering  individuals’  or  groups’  past  interactions  early  can  often  mitigate  issues. 

-  For  example,  there  is  a  clear  benefit  to  knowing  that  a  scientist  on  your  project 
has  had  very  unfavorable  interactions  with  a  potential  emergent  stakeholder  who 
could  toss  a  wrench  in  the  next  phase  of  the  project. 

-  This  prior  knowledge  empowers  you  to  stave  off  potentially  damaging  issues. 
Carefully  examining  the  stakeholders  for  friendship-enabled  work  transactions 
can  guide  the  crafting  of  marketing  proposals  and  requirements. 


[11  Hedrick  Smith,  The  Power  Game:  How  Washington  Works  page  xxii 
[21  Kilduff  and  Tsai,  Social  Networks  and  Organizations  (2003)  page  26-34 


Output-  Understanding  Social  Networks:  Structural  Holes  ♦ 


Sometimes  those  who  refuse  to  cooperate  actually  have  valuable  knowledge  or  abilities; 

they  may  even  be  indispensable  to  your  success. 

•  The  concept  of  structural  holes  is  simply  explained  as  the  absence  or  severe 
shortage  of  ties  between  groups  in  a  network. 

•  This  is  also  the  area  in  which  systems  engineers  can  leverage  when  doing 
requirement  analysis  and  definition  with  the  stakeholders,  including  customers. 

•  Identifying  structural  holes  yields  the  benefits  of  increasing  collaboration  by  furthering 
information  sharing  and  dissemination. 

•  It  may  be  that  by  connecting  two  otherwise  stovepiped  groups,  the  systems  engineer 
can  create  more  stable  requirements  as  groups  converge  to  a  shared  situational 
awareness  (of  course  the  opposite  could  easily  be  the  case)J2l 


[11  Eliza  G.C.  Collins  and  Mary  Anne  Devanna,  The  New  Portable  MBA  page  33 
[21  Kilduff  and  Tsai,  Social  Networks  and  Organizations  (2003)  page  26-34 


If  I  had  eight  hours  to  chop  down  a  tree,  I’d  spend  six  sharpening  my  ax.  -  Abraham 

Lincoln 

•  The  current  systems  engineering  practices  are  not  sufficient  in  situations  where  the 
social  context  dominates  the  environment. 

•  Unfortunately,  the  trend  in  systems  engineering  is  driving  towards  the  need  to 
anticipate  and  manage  issues  arising  from  diverse  and  competing  constituencies  in  a 
complex  environment. 

•  An  important  element  for  successful  systems  engineering  is  the  incorporation  of 
social  network  analysis  into  the  systems  engineering  process  and  in  effect,  using  this 
information  to  engineer  social  success  in  addition  to  technical  success. 


It  is  time  to  formalize  what  successful  systems  engineers  have  been  doing  guided 
by  their  intuition  and  in  an  ad  hoc  manner. 


Additional  Slides 


Requirement  Definition 


♦ 


•  As  previously  mentioned  by  only  simple  inspection,  the  seven  problem  areas  seem  to 
have  a  strong  social  focus.  These  reasons  are  a  product  of  the  social  interaction  of 
the  individuals  involved.  Hence,  it  is  possible  to  correlate  the  problem  areas  in 
systems  engineering  with  the  properties  of  wicked  problems. 

•  Relevant  correlations:  Requirement  articulation 

-  Property  1 :  There  is  no  definitive  formulation  of  a  wicked  problem. 

-  Property  7:  Every  wicked  problem  is  essentially  unique 

-  Property  6:  Wicked  problems  do  not  have  an  enumerable  (or  an  exhaustively 
describable)  set  of  potential  solutions,  nor  is  there  a  well-described  set  of 
permissible  operations  that  may  be  incorporated  into  the  plan 

-  Property  8:  Every  wicked  problem  can  be  considered  to  be  a  symptom  of  another 
problem 

-  Property  5:  Every  solution  to  a  wicked  problem  is  a  “one-shot  operation”; 
because  there  is  no  opportunity  to  learn  by  trial-and-error,  every  attempt  counts 
significantly 


A  problem  area  may  actually  indicate  the  presence  of  a  “wicked”  or 
social  aspect  of  system  development  and  implementation. 


